Summary of Contents for Nielsen ID3
- Page 1 ID3 Tag Validator Application User Guide Release 1.8 Revision F Nielsen Confidential...
- Page 2 Copyright © 2017 The Nielsen Company (US), LLC. All rights reserved. Nielsen and the Nielsen Logo are trademarks or registered trademarks of The Nielsen Company (US), LLC. Other company names, products and services may be trademarks or registered trademarks of their respective companies.
-
Page 3: Table Of Contents
ID3 Tag Validator Application User Manual Contents Overview ......................................6 Purpose ..................................6 Customer Support ................................6 Intended Audience ................................7 Terminology ..................................7 Related Documents ................................7 Using Validator in the Transcoder Certification Process ......................8 Pre-Validation Checklist ................................. 9 Running the Validator Application .............................. - Page 4 ID3 Tag Validator Application User Manual Test 7 Result: CID Timestamp Test ........................ 27 Test 8 Result: CID Timestamp (TS) Offset Test ....................28 Test 9 Result: Breakout Type Test ........................29 Test 10 Result: Watermark Time Test ......................30 Test 11 Result: Watermark Level Test ......................
- Page 5 Improper Set-up of MPEG Multiplexer ..........................58 Background ..............................58 Symptoms ............................... 58 Suggestions for Follow-up ..........................58 ID3 Tags Out-of-Sync with Corresponding Audio ......................59 Background ..............................59 Symptoms ............................... 59 Suggestions for Follow-up ..........................59 Missing or Invalid INFO Tag Settings ..........................59 Background ..............................
- Page 6 ID3 Tag Validator Application User Manual Table 2 – -p Option Settings ................................14 Table 3 – Default Output Files ................................ 16 Table 4 – Interval in Seconds ................................. 19 Table 5 – NAES 2 EDU Description ..............................47 Table 6 – NAES 6 EDU Description ..............................48 Table 7 –Comma-Delimited Tag Listing Imported into Microsoft Excel with Added Headings ............
-
Page 7: Overview
Process an HLS stream with the m3u8 index and transport segments zipped into a file (option –m) • Process a file of encrypted ID3 tags such as the output of the PCM-to-ID3 SDK sample application or the log file of a mobile device that processes Nielsen ID3 tags (option –r) •... -
Page 8: Intended Audience
Terminology To properly interpret the output of Validator, you should be familiar with the basic components of the Nielsen ID3 tag. For an in-depth description of a typical ID3 tag, see “Detailed Tag Listing – DATA Tag” on page 43. -
Page 9: Using Validator In The Transcoder Certification Process
The emsg boxes should be attached to audio segments. Validator is the primary tool used in evaluating the files that you deliver to Nielsen for analysis. Before delivering the test files to Nielsen, you may use Validator to process the files yourself. -
Page 10: Pre-Validation Checklist
ID3 Tag Validator Application User Manual Pre-Validation Checklist A common source of tag-generator errors is the misinterpretation or omission of one or more of the processes listed in Table 1. To avoid these mistakes and save yourself some time, review these steps. - Page 11 SDK (step 4) must be Your application inserts the entire and signaling. The included, without alteration, Nielsen ID3 tag in an emsg box. DASHEventMessageBox in the emsg box. Eliminating scheme_id_uri should be set to or modifying any portion of...
-
Page 12: Figure 1 - Tagging Process
Figure 1 is a visual representation of the tagging process for HLS and MPEG-DASH Streams. The green boxes represent the functionality your application provides. The blue box represents functionality that the Nielsen PCM-to-ID3 SDK provides. Figure 1 – Tagging Process Release 1.8 Revision F... -
Page 13: Running The Validator Application
NielsenId3TagValidator -r <full path name of file holding the ID3 tags> -p 6 Note Validator locates and analyzes all ID3 Tags in the log file and ignores all text entries between tags. for a transport stream with Nielsen ID3 tags “muxed” into it: –t... - Page 14 • For a single file holding a set of MPEG-DASH segments, where some of the segments hold emsg boxes containing Nielsen ID3 tags, you can use the command as follows: NielsenId3TagValidator -d <full path name of file> -p 6 •...
-
Page 15: Optional Parameters
Lists the sequence number and a helpful error message for each Where applicable, lists the structure and contents of the transport stream • reports all fields of the decrypted ID3 tag with multiple lines per Detailed Tag Listing: • summarizes content of a single ID3 tag in a single line Comma-Delimited Tag Listing: with comma-delimited fields. -
Page 16: Program Number Specifier (-N)
ID3 tags. Use the option to specify the program number of the program with the ID3 tags that Validator should analyze. For example, -n 2 instructs Validator to report only on ID3 tags associated with program 2 in a multi-program transport stream. -
Page 17: Validator Output: Summary Report
ID3 tag stream Do the following: To determine whether the source ID3 tag stream contains errors, review the Summary Report. The code labeled as Figure 2 below shows an example of a Summary Report generated when the application was run with the -m option. The Summary Report... -
Page 18: File Header
MPEG - 2 : Tag Time Test: Pass <0/0/6> MPEG - 2 : PES Packet Header Test: Pass <109/81/0> MPEG - 2 : Nielsen Owner ID Test: Pass <109/1/0> MPEG - 2 : Missing PES PTS Test: Pass <0/0/6> MPEG - 2 : PES Header Test: Pass <0/0/6>... -
Page 19: Sid List
The SID List identifies the PC (Program Content) and FD (Final Distributor) SIDs (source identifiers) that Validator found in the ID3 tags in the analyzed content. Validator lists only the SIDs that it associates with the PC or FD code. Due to bit errors during the decoding process, invalid SIDs may appear in the SID List, but their counts are significantly less than those of valid SIDs. -
Page 20: Examples
ID3 Tag Validator Application User Manual 9151: NAES 2 count = 160, NAES 6 count = 166 FD SID 9151: NAES 2 count = 5 , NAES 6 count = 0 PC SID 9051: NAES 2 count = 110 , NAES 6 count = 120 PC SID Figure 4 –... -
Page 21: Info Tag Listing
PC or FD watermarks, the ID3 tag inserter may be thwarted in its ability to generate ID3 tags. If no watermarks appear for the entire 10-second span of an ID3 tag, no ID3 tag is generated for that 10-second period. Therefore, a reduced SID count in the SID List of the Summary Report may well be reflected in a reduced tally of DATA tags. -
Page 22: Test Results
Validator must find at least 237 DATA tags in the stream. ID3 Tag: DATA Tag Count: Pass <310/237/0> Possible Causes of Failure The audio stream fed to the PCM-to-ID3 SDK contained fewer watermarks than expected because of one or more of the following factors: • The audio is silent. -
Page 23: Test 2 Result: Tag Format Test
When Validator is run with the -d, -df, or -dr option (used with MPEG-DASH), Validator confirms that the full 20 bytes of ID3 tag headers and the final two ID3 tag characters (null character, followed by the ASCII character A) are included in the emsg box. In other words, all 271 bytes of the ID3 tag must be included, unaltered, in the emsg box. -
Page 24: Test 3 Result: Sequence Number Test
-51: Tag Error: Detected corrupted, partial, or non-Nielsen tag. -51: Tag error: Separators missing. -3: Emsg Error: Missing private-data byte at end of frame. -30: Emsg Error: Missing ID3 Tag headers. Reported up to 5 times only. Test 3 Result: Sequence Number Test Format <number of sequence-number errors / 0 /... -
Page 25: Test 4 Result: Time Gap Test
If the tags were extracted from an MPEG-2 transport stream or an MPEG-DASH stream, the multiplexer or packager could have failed to include the ID3 tags in the PES. In investigating tag-sequence errors, you must consider all possible points along the distribution chain where tags potentially could be dropped. -
Page 26: Test 5 Result: Tag Duration Test
If a section of the audio stream does not contain watermarks for 10 seconds or longer, the PCM-to-ID3 SDK does not generate an ID3 tag for that segment. A time gap between consecutive tags would exist. The sequence numbers would be consecutive, however, which indicates that no generated tags were dropped. -
Page 27: Test 6 Result: Cid Sid Test
Maximum Tolerated declared Meaning Each ID3 tag includes a PC CID (PC Content ID) and an FD CID. The CID has two components: a Nielsen SID and a Nielsen timestamp, both derived from recent Nielsen watermarks. Consequently, the following must occur: •... -
Page 28: Test 7 Result: Cid Timestamp Test
• More than 10 seconds have transpired without any watermarks. In this case, the CID SID reverts to 0 as the PCM-to-ID3 SDK aligns to the next group of watermarks. • The watermark decoder may not have properly decoded the SID in the source watermark possibly because the audio is difficult to decode due to compression or improper resampling. -
Page 29: Test 8 Result: Cid Timestamp (Ts) Offset Test
CID timestamp plus the tag offset. For example, if the CID timestamp is 3 a.m. and the CID timestamp offset is 3600, Validator should report watermarks in the ID3 tag within 10 seconds of 4 a.m., which is 3 a.m. + 3600 seconds. -
Page 30: Test 9 Result: Breakout Type Test
CID timestamp offset errors directly relates to the number of watermark time warnings. Occasional decoding errors are to be expected and, after careful examination of the log file and the ID3 tag listing, will be excused in most cases. (See “Test 10 Result: Watermark Time Test” on page 30.) •... -
Page 31: Test 10 Result: Watermark Time Test
(or remain the same) at a pace that is not synchronized with the running transport-stream or ID3 tag “clock.” It also keeps track of changes in daylight saving time (DST) flags. Any discrepancies between the expected watermark time (or DST flag) and the actual setting are tallied as watermark time warnings. -
Page 32: Test 11 Result: Watermark Level Test
ID3 Tag Validator Application User Manual Test 11 Result: Watermark Level Test Format <number of watermark level errors / 0 / Format maximum tolerated> Number of Errors Number of errors detected Not used Maximum Tolerated Maximum number of errors tolerated before failure is declared Meaning Each watermark is labeled with a specific SID and a specific level. -
Page 33: Test 12 Result: Info Tag Count
ID3 Tag Validator Application User Manual Test 12 Result: INFO Tag Count Format Summary Report Format <count of INFO tags/ minimum required / 0> Count Number of INFO tags found in the stream Minimum Required Minimum number of INFO tags required... - Page 34 • If the PCM-to-ID3 SDK releases a DATA tag and an INFO tag immediately after your application invokes a call to InputAudioData(), your application may be dropping the INFO tag. Make certain to transmit all ID3 tags that are generated after InputAudioData() returns.
-
Page 35: Tests 14 And 15: Info Tag Cid And Offset Tests
ID3 Tag Validator Application User Manual Meaning The INFO Tag Interval test reports the number of times an INFO tag appeared too soon or too late, in comparison to the previous INFO tag. Validator may report a tag as missing at 310 seconds and may report the next tag as arriving too soon. -
Page 36: Test 16 Through 21 Results: Info Tag Tests
• Audio CODEC (no longer checked) If a setting is missing or invalid, Validator declares an error. The PCM-to-ID3 SDK Developer’s Guide identifies the length and format of each field. Because all INFO tags are identical, the SDK does not deliver a message for each observed error. -
Page 37: Test 22 Result: Tag Placement Check
If the incoming content is a transport stream that includes a watermarked AC-3 elementary stream, the tag-placement error field keeps track of the number of times that an ID3 tag appears to be out of sync with the audio content that it represents. Note If the incoming stream is not a transport stream or the test is not conducted, the status is set to “No Status.”... -
Page 38: Test 23 Result: Tag-Time Test
Pass <0/0/7> Possible Causes of Failure The PCM-to-ID3 SDK requires that the calling application maintain a system clock by correctly setting the currentTime argument to the InputAudioData() function. The currentTime value must advance 1 second for each second of audio content processed. -
Page 39: Test 24 Result: Pes Packet Header Check
The most likely cause of a tag-time error is poor multiplexing. For proper multiplexing to occur, the PTS of each ID3 DATA Tag PES header must be set to the PTS of the corresponding audio PES header. If the PTS values are properly set, the transport- stream multiplexer should properly place the tags. -
Page 40: Test 25 Result: Pes Owner-Id Check
When Validator finds the string “www.nielsen.com” near the beginning of a PES packet, it logs the event. The purpose of this test is to ascertain that the ID3 tag metadata elementary stream is, indeed, a Nielsen ID3 tag stream. Validator must only find one Nielsen URL to confirm this fact. -
Page 41: Test 27 Result: Pes Header Test
HTTP Live Streaming.” See “Related Documents” on page 7. Corresponding Log Message If the PES packet stream ID is incorrectly set or if the PES packet size of a Nielsen ID3 PES packet is set to 0, one of the following error messages is logged:... -
Page 42: Test 28 Result: Pes Incomplete Tag Test
Maximum Tolerated failure is declared Meaning If the size of the payload in the Nielsen ID3 metadata stream is not large enough to hold a complete Nielsen ID3 tag, Validator reports an error. An incomplete payload cannot be decoded. Each Nielsen ID3 tag spans two transport packets. If the size of the payload (as specified in the PES header) is correct, but the second transport packet contributing to the payload is missing, Validator declares an incomplete-tag error. -
Page 43: Tests 29 And 30 Result: Pmt Elementary Stream And Descriptor Checks
For an MPEG-2 decoder to detect a Nielsen ID3 tag elementary stream, the decoder must find an entry for a stream of type 0x15 followed by an MPEG-2 ID3 tag descriptor in the PMT elementary-stream loop. Each time that the decoder finds a stream of type 0x15 in the loop, Validator counts it as a metadata PES. -
Page 44: Detailed Tag Listing - Data Tag
If you set the -p command-line option to 2 or 6, Validator generates a Detailed Tag Listing and reports all fields of each Nielsen ID3 tag that it detects. Remember that there are two types of ID3 tags: DATA and INFO. Ideally, DATA tags are delivered at 10-second intervals and INFO tags are delivered at 5-minute intervals. -
Page 45: Tag Number
ID3 tag was packaged and released. If this value is not correctly set, the Nielsen ID3 tags may fall out of sync with the audio stream, resulting in improper crediting. The PTS should advance at roughly the same rate as the First and Last tag times. -
Page 46: Seconds Since Fd Day-Start (Fd Cid Offset)
Under ideal conditions, the FD CID SID identifies the SID of the last NAES 2 FD watermark listed among the EDUs of the current ID3 tag. In the absence of NAES 2 watermarks, special rules apply to the selection of the SID based on NAES 6 watermarks. -
Page 47: Sequence Number (Seqno)
The PCM-to-ID3 SDK receives an audio stream from your application, summarizes the contents of the watermark by creating an EDU, and stores the EDU in an ID3 tag. This means that there is a “watermark SID” (the SID in the source watermark) and the EDU is a representation of that SID. -
Page 48: Naes 2 Edu
Distribution Watermark level is Program Content Sample 5225 Nielsen admin use only Offset PCM-to-ID3 SDK decoded this watermark 1 second after the start of this ID3 tag Time String 09/15/2015 19:00:53 See TS/TIC above. Time Type TIME Timestamp of the EDU (see TS/TIC above). -
Page 49: Naes 6 Edu
NAES encoder generated the watermark during daylight saving time Distribution Watermark level is Final Distributor 1 Sample Nielsen admin use only Offset PCM-to-ID3 SDK decoded this watermark 6 seconds after the start of this ID3 tag Time string 09/15/2015 19:00:58 See TS/TIC above. Time type TIME Timestamp of the EDU (see TS/TIC above). -
Page 50: Evaluating Edus For Inclusion In The Summary Sid List
PCM-to-ID3 SDK. When processing a stream of ID3 tags, Validator frequently identifies the first few watermarks with a specific SID as having an “undefined” level (marked “UK” in the EDU line). -
Page 51: Comma-Delimited Tag Listing
DATA tags. A missing sequence number in the detailed tag listing usually is an indication that the ID3 tag was dropped or delivered out of order sometime after the PCM-to-ID3 SDK generated it. It is acceptable for INFO Tags to be delivered out of order. - Page 52 The Tag Time corresponds to the First tag time, which identifies the time at which the PCM-to-ID3 SDK began to collect watermarks for the current Nielsen ID3 tag. The tag times of consecutive DATA rows should be no more than 12 seconds apart, assuming that the processed audio contains uninterrupted Nielsen watermarks.
- Page 53 Under ideal conditions, the PC CID SID identifies the SID of the last PC watermark listed among the EDUs of the current ID3 tag. It takes the PCM-to-ID3 SDK 10 seconds or more to “latch onto” the proper PC SID. As a result, the PC CID may be inaccurate (or set to 0) for 10–20 seconds after a SID changes or disappears.
- Page 54 Under ideal conditions, the FD CID SID identifies the SID of the last FD watermark listed among the EDUs of the current ID3 tag. It takes the PCM-to-ID3 SDK 10 seconds or more to “latch onto” the proper FD SID. As a result, the FD CID may be inaccurate (or set to 0) for 10–20 seconds after a SID changes or disappears.
- Page 55 ID3 Tag Validator Application User Manual FD CID SID mismatch; FD CID TS inconsistency 1444645071 9020 2800909039 9020 2800851248 57791 26516 1582259560 9120 2800851248 57790 The FD CID Timestamp is the decrypted timestamp component of the 24-byte PC Content ID.
- Page 56 PTS of the last audio packet presented to the PCM-to-ID3 SDK before the ID3 tag was packaged and released. If this value is not set correctly, the Nielsen ID3 tags may fall out of sync with the audio stream and result in improper crediting.
-
Page 57: Troubleshooting Tips
Tag Time Not Advancing Properly Background Each ID3 tag has a start time and an end time. Ideally, the difference between the start and end times should be 10 seconds. If the tag fills with 10 or more EDUs, the tag- duration is legitimately less than 10 seconds. -
Page 58: Processing Set To Incorrect Mode (Streaming Or Mdi)
The PCM-to-ID3 SDK uses a callback function to deliver ID3 tags to the calling application. The call to InputAudioData() does not have a one-to-one relationship with the SDK response in delivering an ID3 tag. In fact, typically many calls to InputAudioData() take place between callback reports of ID3 tags that are ready for delivery. -
Page 59: Suggestions For Follow-Up
If you do not properly set up the PMT of the transport stream to indicate the presence of the ID3 tag PES in the transport stream, neither Validator nor the mobile meters see any Nielsen ID3 tags. Your application must follow the rules—especially the rules for PMT and PES—in the “Timed Metadata for HTTP Live Streaming, Networking &... -
Page 60: Id3 Tags Out-Of-Sync With Corresponding Audio
When Validator processes transport stream input that has both watermarked AC-3 audio ID3 tags, it reports a synchronization error if the tags are separated by more than 15 seconds from the corresponding audio as shown in this example: MPEG-2: Missing PES PTS Test: Fail <110/0/4>... -
Page 61: Missing Or Invalid Breakout Type Setting
“Unexpected SIDs” refers to SIDs that are valid but are not expected to appear. (More specifically, there is nothing wrong with the SID itself.) The ID3 tags should reflect the Nielsen SID assigned to the content that is being viewed. The Summary Report lists the... -
Page 62: Checking Tag Synchronization In Mpeg-Dash Content
Compare the SIDs and times in the watermarks in NACAT to the SIDs and times of the EDUs identified in step 2. For watermarks with matching SIDs, the timecodes in NACAT should fall within 15 to 20 seconds of the timecodes in the ID3 tag EDUs. Release 1.8 Revision F... -
Page 63: Glossary
Content CID) and an FD CID (Final Distributor CID). The PC CID represents the PC EDUs included in the current ID3 tag. The FD CID represents the FD EDUs included in the current ID3 tag. The timestamp for both CIDs is rolled back to the most recently “crossed”... - Page 64 EDU offset The EDU offset is the eighth field in each EDU line of the ID3 tag listing in the _tags.txt file that Validator generates. The EDU offset specifies when, in relation to the ID3 tag start time, the source watermark was decoded.
- Page 65 PC CID timestamp of the tag. The sum of the PC CID timestamp and the PC Offset should be within 15 seconds of the most recent PC EDU in the current ID3 tag. In the _tags.txt file, the FD offset is labeled “Seconds Since PC Day-Start.”...
- Page 66 EDU was derived. tag number Indicates the ordinal position of the current ID3 tag among all of the tags in the detailed tag listing. It reflects the order in which Validator extracted the ID3 tags from the metadata PES. tag time See ID3 tag time.
- Page 67 (or content) to which the viewing should be credited. watermark SID The SID in the source watermark. Note that the EDU a representation of the watermark SID that is stored in an ID3 tag. Release 1.8 Revision F...