lazy decoding #34

chuff · Aug 30, 2023

No description provided.

iabgpp-encoder/src/test/java/com/iab/gpp/encoder/GppModelTest.java

iabgpp-encoder/src/main/java/com/iab/gpp/encoder/GppModel.java

iabgpp-encoder/src/main/java/com/iab/gpp/encoder/base64/TraditionalBase64UrlEncoder.java

Net-burst · Jan 25, 2024

I took the liberty to do a very simple performance comparison between the master and this PR. The test was extremely simple and mostly directed toward measuring the decoding performance and the performance of pulling out the EU TCF string out of GPP string. The test was done using the JMH.

    @Benchmark
    public void decodeOnly(Blackhole blackhole, ConsentString consentString) throws DecodingException, EncodingException {
        blackhole.consume(new GppModel(consentString.consent));
    }

    @Benchmark
    public void decodeAndGet(Blackhole blackhole, ConsentString consentString) throws DecodingException, EncodingException {
        GppModel gppModel = new GppModel(consentString.consent);
        blackhole.consume(gppModel.encodeSection(TcfEuV2.ID));
    }

I used 2 different consent strings for this: the one from the Readme.md and a random real GPP string generated by the real CMP in Europe. I'll call those short and long GPP string accordingly.

The current master

Short GPP
Benchmark                     Mode  Cnt       Score     Error  Units
GppLibPerfTest.decodeAndGet  thrpt   25   55050.257 ± 798.702  ops/s
GppLibPerfTest.decodeOnly    thrpt   25  104209.384 ± 690.231  ops/s

Long GPP
Benchmark                     Mode  Cnt     Score   Error  Units
GppLibPerfTest.decodeAndGet  thrpt   25   312.324 ± 0.579  ops/s
GppLibPerfTest.decodeOnly    thrpt   25  2649.314 ± 6.534  ops/s

Please notice the huge performance degradation when transitioning from the example GPP string to a real-world GPP string.

This PR

Short GPP
Benchmark                     Mode  Cnt          Score        Error  Units
GppLibPerfTest.decodeAndGet  thrpt   25     607814.864 ±   1748.729  ops/s
GppLibPerfTest.decodeOnly    thrpt   25  269170680.844 ± 536131.656  ops/s

Long GPP
Benchmark                     Mode  Cnt          Score        Error  Units
GppLibPerfTest.decodeAndGet  thrpt   25     587291.468 ±   1715.360  ops/s
GppLibPerfTest.decodeOnly    thrpt   25  268889768.896 ± 477279.182  ops/s

Pure decoding is meaningless now, as nothing happens before anything is requested from the object. However, the TCF EU extraction performance is much better. And apart from the huge overall performance improvement, the performance degradation with real-world GPP string is gone.

The test was done on an Apple Macbook M2 Pro. It's not scientific in any way though and was done purely for comparison.

Net-burst

LGTM. I didn't review all the logic fully, but I don't see any apparent issues. Existing functionality doesn't seem to be affected and the performance benefit is tremendous for decoding operations.

ym-corey · Feb 12, 2024

Are there any plans to release this sometime soon?

Net-burst · Mar 18, 2024

Any updates on this one? Is there a plan to merge it anytime soon?

SNMY · Jul 20, 2024

iabgpp-encoder/src/main/java/com/iab/gpp/encoder/base64/CompressedBase64UrlEncoder.java

+  @Override
+  protected String pad(String bitString) {
+    char[] chars1 = null;
+    if(bitString.length() % 8 > 0) {


Hi, @iabmayank
We have another question here regarding the GPP encoding and decoding. Why must the length of bitString be a multiple of 8 first? We recently encountered some GPP strings like DBABzw~~1YYN~~BVQVAAAAAg that don't meet this padding logic but can still be decoded via the IAB GPP Encoder / Decoder.
Could you please share your thoughts on this?
Thanks in advance.

* lazy decoding * java 8 compatible gppmodel tests * rename missed multistate usp* methods to us* * lazier decoding * lazier decoding * tests for null and empty string constructor arguments * deprecate multi-state usp* methods * remove deprecated usp methods * encodeSection lazy fix * optimize bitstring padding --------- Co-authored-by: chad <chad@totalium.com>

* initial multi state implementation * multi state * unit tests * exposing id based methods in GppModel * personal data consents fix, reduce multi state subsection id (#10) Co-authored-by: chad <chad@totalium.com> * 3.0.0 (#11) * personal data consents fix, reduce multi state subsection id * scm * [maven-release-plugin] prepare release iabgpp-core-3.0.0 * scm * scm * scm * scm * 3.0.0 * 3.0.1-SNAPSHOT Co-authored-by: chad <chad@totalium.com> * fix multi state name, add unit tests (#13) * personal data consents fix, reduce multi state subsection id * scm * [maven-release-plugin] prepare release iabgpp-core-3.0.0 * scm * scm * scm * scm * 3.0.0 * 3.0.1-SNAPSHOT * fix section names and add unit tests Co-authored-by: chad <chad@totalium.com> * 3.0.1 (#14) * personal data consents fix, reduce multi state subsection id * scm * [maven-release-plugin] prepare release iabgpp-core-3.0.0 * scm * scm * scm * scm * 3.0.0 * 3.0.1-SNAPSHOT * fix section names and add unit tests * 3.0.1 * 3.1.0-SNAPSHOT * cleanup * fix groupId in README * fix groupId in README * fix groupId in README Co-authored-by: chad <chad@totalium.com> * Convenience Methods (#15) * personal data consents fix, reduce multi state subsection id * scm * [maven-release-plugin] prepare release iabgpp-core-3.0.0 * scm * scm * scm * scm * 3.0.0 * 3.0.1-SNAPSHOT * fix section names and add unit tests * 3.0.1 * 3.1.0-SNAPSHOT * cleanup * fix groupId in README * fix groupId in README * fix groupId in README * convenience methods * 3.0.2 * 3.1.0-SNAPSHOT Co-authored-by: chad <chad@totalium.com> * full list of fields to readme (#16) * personal data consents fix, reduce multi state subsection id * scm * [maven-release-plugin] prepare release iabgpp-core-3.0.0 * scm * scm * scm * scm * 3.0.0 * 3.0.1-SNAPSHOT * fix section names and add unit tests * 3.0.1 * 3.1.0-SNAPSHOT * cleanup * fix groupId in README * fix groupId in README * fix groupId in README * convenience methods * 3.0.2 * 3.1.0-SNAPSHOT * full list of fields to readme * rename array to list in readme Co-authored-by: chad <chad@totalium.com> * tcfeuv2 backwards compatible (#19) * Exception handling * compressed base64urlencoder for all non-tcfeuv2 segments * Revert uspv1 to original encoding * Rename tcfcav2 to tcfcav1 * update readme for tcfcav2 -> tcfcav1 and uspv1 changes * optional gpc segments * update readme with GpcSegmentIncluded fields * fix vendor list encoding / decoding * formatting * 3.0.3 * version bump Co-authored-by: chad <chad@totalium.com> * Update dependencies. Fix bitfield cast. (#20) * Exception handling * compressed base64urlencoder for all non-tcfeuv2 segments * Revert uspv1 to original encoding * Rename tcfcav2 to tcfcav1 * update readme for tcfcav2 -> tcfcav1 and uspv1 changes * optional gpc segments * update readme with GpcSegmentIncluded fields * fix vendor list encoding / decoding * formatting * 3.0.3 * version bump * fix bitfield data type * update jackson version * update junit * 3.0.4 * 3.0.5-SNAPSHOT * 3.0.5 * 3.0.6-SNAPSHOT * 3.0.6 * 3.0.7-SNAPSHOT * 3.0.7 * 3.0.8-SNAPSHOT Co-authored-by: chad <chad@totalium.com> * Dedupe and sort lists. Added InvalidFieldError. (#24) * fix optimized fix range decoder * fix optimized fibonacci fix range decoder * 3.0.8 * 3.0.9-SNAPSHOT * 3.0.9 * 3.0.10-SNAPSHOT * dedup and sort lists * formatting * 3.0.10 * 3.0.11-SNAPSHOT * 3.0.12-SNAPSHOT * 3.0.11-SNAPSHOT * 3.0.10 * 3.0.11-SNAPSHOT * 3.0.10 * 3.0.11-SNAPSHOT * cleanup Co-authored-by: chad <chad@totalium.com> * issue 26: wrong version number for TcfCaV1 (#27) Co-authored-by: Guofang Li <guofang.li@thetradedesk.com> * Removes duplicate section in the GppModel.decodeSection if tree as UspV1 was in the list twice. (#30) * Updating GVL code as V3 and adding some comments (#32) * #29 #26 * Removes duplicate section in the GppModel.decodeSection * Rename multistate usp* to us* * fix tcfcav1 policy version * vendor list 2.2 * lazy decoding * cleanup lazy decoding * Updating GVL code as V3 and adding some comments; also removing GPPModel LazyDecoding feature; also keeping the TCF policy version as 2 for Canada. * 3.1.0 * 3.1.1-SNAPSHOT --------- Co-authored-by: chad <chad@totalium.com> Co-authored-by: srini81 <srinivas.bhagavatula@yahooinc.com> * rename missed multistate usp* methods to us* (#35) Co-authored-by: chad <chad@totalium.com> * Adding a new field in Vendor for "impConsPurposes" as the GVL for Canada has changed to include this field in the place of "legIntPurposes". (#37) Both these fields will be set to Optional in order to handle both Canada and EU use cases. Co-authored-by: srini81 <srinivas.bhagavatula@yahooinc.com> * 3.1.1 * 3.1.2-SNAPSHOT * Version Bump (#38) * 3.1.1 * 3.1.2-SNAPSHOT --------- Co-authored-by: chad <chad@totalium.com> * lazy decoding (#34) * lazy decoding * java 8 compatible gppmodel tests * rename missed multistate usp* methods to us* * lazier decoding * lazier decoding * tests for null and empty string constructor arguments * deprecate multi-state usp* methods * remove deprecated usp methods * encodeSection lazy fix * optimize bitstring padding --------- Co-authored-by: chad <chad@totalium.com> * Update README.md (#46) Fix incorrect client-side API prefixes. https://github.com/InteractiveAdvertisingBureau/Global-Privacy-Platform/blob/main/Sections/Section%20Information.md is the authority. * Validation (#36) * lazy decoding * java 8 compatible gppmodel tests * rename missed multistate usp* methods to us* * lazier decoding * lazier decoding * tests for null and empty string constructor arguments * validation * fix typo * remove redundant validate call * default validate * remove empty validate method from header core segment * fix usct validator * deprecate multi-state usp* methods * remove deprecated usp methods * cleanup validators * encodeSection lazy fix --------- Co-authored-by: chad <chad@totalium.com> * tcfca pub restrictions and disclosed vendors (#39) * rename missed multistate usp* methods to us* * 3.1.1 * 3.1.2-SNAPSHOT * tcfca publisher restrictions and disclosed vendors * deprecate multi-state usp* methods * substring error handling * remove deprecated usp methods * Update README * tcfeu pub restrictions fix * pub restrictions getters * tcfeu pub restirctions fix * cleanup * pub restrictions fix --------- Co-authored-by: chad <chad@totalium.com> * 3.2.0 * 3.2.1-SNAPSHOT * Versioning (#47) * 3.1.1 * 3.1.2-SNAPSHOT * 3.2.0 * 3.2.1-SNAPSHOT --------- Co-authored-by: chad <chad@totalium.com> * Remove multistate version (#60) * lazy decoding * java 8 compatible gppmodel tests * rename missed multistate usp* methods to us* * lazier decoding * lazier decoding * tests for null and empty string constructor arguments * validation * fix typo * remove redundant validate call * default validate * remove empty validate method from header core segment * fix usct validator * 3.1.1 * 3.1.2-SNAPSHOT * tcfca publisher restrictions and disclosed vendors * deprecate multi-state usp* methods * substring error handling * remove deprecated usp methods * remove deprecated usp methods * remove deprecated usp methods * cleanup validators * Better decoding exception messaging * remove unused classes * Update README * add support for the old headerless tcfeuv2 strings * encodeSection fix * encodeSection lazy fix * tcfeu pub restrictions fix * pub restrictions getters * tcfeu pub restirctions fix * cleanup * pub restrictions fix * optimize bitstring padding * 3.2.0 * 3.2.1-SNAPSHOT * fl mt or tx * remove version from multistate resources --------- Co-authored-by: chad <chad@totalium.com> * Post release version bump (#64) * 3.2.1 * 3.2.2-SNAPSHOT --------- Co-authored-by: Chad Huff <chad@totalium.com> * 3.2.2 * 3.2.3-SNAPSHOT * Tx fl or mt (#57) * lazy decoding * java 8 compatible gppmodel tests * rename missed multistate usp* methods to us* * lazier decoding * lazier decoding * tests for null and empty string constructor arguments * validation * fix typo * remove redundant validate call * default validate * remove empty validate method from header core segment * fix usct validator * 3.1.1 * 3.1.2-SNAPSHOT * tcfca publisher restrictions and disclosed vendors * deprecate multi-state usp* methods * substring error handling * remove deprecated usp methods * remove deprecated usp methods * remove deprecated usp methods * cleanup validators * Better decoding exception messaging * remove unused classes * Update README * add support for the old headerless tcfeuv2 strings * encodeSection fix * encodeSection lazy fix * tcfeu pub restrictions fix * pub restrictions getters * tcfeu pub restirctions fix * cleanup * pub restrictions fix * optimize bitstring padding * 3.2.0 * 3.2.1-SNAPSHOT * fl mt or tx --------- Co-authored-by: chad <chad@totalium.com> * Update AbstractLazilyEncodableSection.java (#61) * DE IA NE NH NJ TN MSPA USNAT (#65) * DE IA NE NH NJ TN * remove validation * usnat backwards compatibility --------- Co-authored-by: Chad Huff <chad@totalium.com> * remove validation (#66) * remove validation * fix lazy decoding in decodeSection --------- Co-authored-by: Chad Huff <chad@totalium.com> * 3.2.3 * 3.2.4-SNAPSHOT * Added GitHub Actions workflow file for iabgpp-java version release * Fail if gpp string doesnt start with DB * fix merge conflict --------- Co-authored-by: Mayank Mishra <mayank@iabtechlab.com> Co-authored-by: chad <chad@totalium.com> Co-authored-by: Guofang Li <guofang.li@live.com> Co-authored-by: Guofang Li <guofang.li@thetradedesk.com> Co-authored-by: Bryan DeLong <ananon@gmail.com> Co-authored-by: srini81 <srinivas.bhagavatula@yahooinc.com> Co-authored-by: Srinivas Bhagavatula <srini.cancerian@gmail.com> Co-authored-by: Daniël Hoeksema <37441336+danielsao@users.noreply.github.com> Co-authored-by: iabsxhxl <sahil.kukreja@aitglobalinc.com> Co-authored-by: naveenjr7 <cnaveenchidambaram@gmail.com> Co-authored-by: aitsxhxl <111370634+aitsxhxl@users.noreply.github.com>

lazy decoding

8b84427

srinivas81 reviewed Aug 31, 2023

View reviewed changes

iabgpp-encoder/src/test/java/com/iab/gpp/encoder/GppModelTest.java Outdated Show resolved Hide resolved

java 8 compatible gppmodel tests

89c5567

srinivas81 reviewed Aug 31, 2023

View reviewed changes

iabgpp-encoder/src/main/java/com/iab/gpp/encoder/GppModel.java Outdated Show resolved Hide resolved

srinivas81 reviewed Aug 31, 2023

View reviewed changes

iabgpp-encoder/src/main/java/com/iab/gpp/encoder/GppModel.java Outdated Show resolved Hide resolved

chad added 9 commits September 2, 2023 18:54

rename missed multistate usp* methods to us*

1c4ae8a

lazier decoding

2b159f4

lazier decoding

4a516b9

tests for null and empty string constructor arguments

6eccd6d

deprecate multi-state usp* methods

8af6c08

remove deprecated usp methods

46323b7

Merge branch 'multistate-usp-to-us' into lazy-decoding

cdffeab

Merge remote-tracking branch 'upstream/master' into lazy-decoding

9ff3562

encodeSection lazy fix

4f182f8

simontrasler reviewed Nov 13, 2023

View reviewed changes

iabgpp-encoder/src/main/java/com/iab/gpp/encoder/base64/TraditionalBase64UrlEncoder.java Outdated Show resolved Hide resolved

optimize bitstring padding

7ca0288

simontrasler approved these changes Dec 20, 2023

View reviewed changes

Net-burst approved these changes Jan 31, 2024

View reviewed changes

iabmayank merged commit 60eb664 into IABTechLab:master Apr 9, 2024

patmmccann mentioned this pull request Apr 19, 2024

High CPU Consumption #25

Open

SNMY reviewed Jul 20, 2024

View reviewed changes

yuzawa-san mentioned this pull request Mar 5, 2025

java.lang.OutOfMemoryError: Java heap space when decoding non GPP string #44

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

lazy decoding #34

lazy decoding #34

Uh oh!

chuff commented Aug 30, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Net-burst commented Jan 25, 2024 •

edited

Loading

Uh oh!

Net-burst left a comment

Uh oh!

ym-corey commented Feb 12, 2024

Uh oh!

Net-burst commented Mar 18, 2024

Uh oh!

SNMY Jul 20, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Search code, repositories, users, issues, pull requests...

lazy decoding #34

lazy decoding #34

Uh oh!

Conversation

chuff commented Aug 30, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Net-burst commented Jan 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

The current master

This PR

Uh oh!

Net-burst left a comment

Choose a reason for hiding this comment

Uh oh!

ym-corey commented Feb 12, 2024

Uh oh!

Net-burst commented Mar 18, 2024

Uh oh!

SNMY Jul 20, 2024

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Net-burst commented Jan 25, 2024 •

edited

Loading