Headline Testing and Optimization

Your headlines are the first impression your content makes. Headline Testing & Optimization empowers Parse.ly customers to take the guesswork out of crafting the perfect headline. By displaying multiple headline variants to different visitors and automatically identifying which one performs best, this feature helps maximize engagement through higher click-through rates (CTR).

With real-time testing powered by smart algorithms, content teams can make confident, data-driven decisions without manual analysis. Whether aiming to increase visibility in search engines or captivate readers on social media, Headline Testing ensures your best headline rises to the top.

Note: This feature is currently in beta. The product is still evolving, so please expect occasional downtimes and potential data loss during this phase.

Headline Testing Demo

Watch a demo on how to create a new test.

Why Headline Testing Matters

Your content competes for attention in a crowded digital landscape. A compelling headline can mean the difference between a reader clicking or scrolling past. Headline Testing gives your team a systematic way to:

• Validate headline ideas based on actual audience behavior.
• Increase the effectiveness of editorial decisions.
• Learn what resonates with your readers to inform future content strategies.

Key Benefits

Our Headline Testing feature provides several significant benefits to help your content perform at its best:

• Test up to 10 headlines simultaneously—1 control and 9 variants—giving your team the freedom to try bold experiments or subtle refinements.
  
• You can boost your click-through rates (CTR) and increase reader engagement by identifying the most effective headlines.
• The feature enables you to make fast, data-driven editorial decisions based on real user behavior.
• The testing focuses on what matters most—understanding and capturing your audience’s attention.
• You can automate the hard part of the headline testing process without requiring manual monitoring or frequent adjustments.
• Run up to 10 headline experiments at the same time, enabling you to continuously optimize multiple stories across your site.
• Each experiment can be configured to run from as little as 15 minutes to as long as 7 days, adapting to the lifecycle and urgency of your content.
  

How It Works

1. First, a separate JavaScript script must be installed on your website. This script is intentionally designed to be as lightweight as possible, separate from the main Parse.ly script, to ensure it loads very quickly and before any page content is displayed. Find installation instructions and options below. 
2. Once the script loads, it already comes preloaded with the list of active headline experiments. The system attempts to locate the control headline attached to each article link on the page.
3. If the control headline is found, the script replaces it with the assigned variant headline for that specific visitor.
4. When the variant headline is displayed the first time, this counts as an impression for that visitor.
5. If the visitor clicks on the headline the first time, it is recorded as a success for that headline variant, and the system tracks this click.
6. The selected headline variant is stored in the visitor’s cookies. This ensures consistency so that the same visitor will continue to see the same headline variant on subsequent visits.
7. The system uses a multi-armed bandit approach, specifically Thompson Sampling with a Beta distribution, to determine which headline should be shown more frequently. For each variant, the algorithm models the probability of success based on observed data:
   • Alpha (α) represents the number of unique clicks on a headline variant under test.
   • Beta (β) represents the number of visitors the headline has been displayed to (impressions) but not clicked.
8. Over time, the algorithm dynamically adjusts the probability of showing each variant, favoring headlines that perform better while still exploring other options to continuously learn.

This entire process happens automatically and quickly, enabling real-time optimization without requiring any manual analysis or adjustments.

Best Practices

Tips for Effective Headline Testing

• Use variety: Try emotional, question-based, and keyword-focused variants.
• Short tests (15–30 min) work well for breaking news.
• Don’t test too many at once if traffic is low—aim for statistical confidence.

Limitations

• We only support headlines inside an anchor tag (<a>) linking to the URL.
• You need to be able to add JS to the <head> of their website.

Access

To access Headline Testing & Optimization:

1. Navigate to the Parse.ly Dashboard.
2. Select a website from the dropdown located at the upper left of the Parse.ly Dashboard.
3. Select “Headline Testing” from the second-level navigation at the top of the screen.

Permissions

By default, only Admins can completely manage tests. Non-admins cannot create/start or end tests.

How to set permissions?

1. Admins can go to the User Settings page: https://dash.parsely.com/<api_key>/settings/users
   • You can also click on the profile icon dropdown button on the top-right corner of the Parse.ly Dash screen, and then select the “Users” option.
2. Select a user using the checkbox on the left-most column.
3. Click on the Edit Permissions button that shows up. It should open up a dialog box like this, where they can check/uncheck the Manage Headline Tests permission for that user and that API Key.

Setup a test

Running a headline experiment in Parse.ly is quick and intuitive. Just follow these six steps:

1. Go to the Headline Testing Dashboard
   Access it from the main navigation in your Parse.ly dashboard.
2. Click on “Create Test”
   You’ll find this button at the top right of the Headline Testing view.
3. Select the Post You Want to Test
   Choose any published post to start testing different headline options.
4. Set the Duration
   Define how long your test should run—anywhere from 15 minutes up to 7 days.
5. Add Headline Variants
   Include up to 10 headlines (1 control + 9 variants). Be bold or subtle—test what matters most.
6. Start the Test
   Launch your experiment. Our system will automatically distribute traffic and begin learning in real time.

Pro tip: You can monitor your test’s performance live and stop it anytime if needed.

AI-Generated Headlines

Parse.ly offers the ability to generate headline suggestions using AI directly within the Headline Testing interface. This feature is designed to spark creativity, accelerate workflows, and help editorial teams test fresh headline variations that align with their story’s tone and intent.

Note: This feature is currently behind a feature flag and may not be available in all accounts. To enable it for your site, please contact support@parsely.com.

How It Works

When configuring a new headline test, users with the feature enabled will see a Write with AI button.

1. Prompt Input
   After clicking the button, you are guided through a short form where you can provide context for the AI to generate relevant suggestions.
   
2. AI Suggestions
   A list of suggested headlines is then displayed. You can:
   
   • Pin your favorite suggestions.
   • Discard unwanted options.
   • Request more suggestions.
     
3. Add to Variants
   Once satisfied, you can add selected AI-generated headlines to the variant list for testing. Each AI-generated variant is flagged in the system, allowing for future analysis of performance compared to human-written options.

Editing the Control Headline

You can now edit the control headline, which is the main headline shown on your site and in your CMS when setting up a headline test. This gives you more flexibility to refine your tests and ensure the right version of the headline is used as the foundation for the experiment.

This feature is helpful when:

• You have different headlines depending on the location of the post (News section, homepage, archive,…)
• You want to adjust the main headline before launching a test.
• The published headline has changed since the test was created.

Note: This feature is currently behind a feature flag and may not be available in all accounts. To enable it for your site, please contact support@parsely.com.

How to enable Headline Testing on your website

Script Installation

One-Line Snippet

Place this script in the <head> before any other scripts but after all <meta /> tags:

<script src="https://experiments.parsely.com/vip-experiments.js?apiKey=XXXXX"></script>

Make sure to add your Site ID as the apiKey to the script’s source. This should be a domain-like value, such as example.com, and not your API Secret! Please ensure to test the script on your staging website first to ensure everything loads correctly and there are no errors in your browser console.

The script is approximately 9kB (gzipped) and is delivered via our global edge network. This implementation loads the script synchronously, ensuring experiments run as often as possible.

To load the script asynchronously, add the async attribute. However, note that experiments may run less frequently if the script executes after the first contentful paint.

Advanced Installation

If the script loads asynchronously, you may experience flickering as headlines are replaced. To minimize this, use the advanced installation method and enable flicker control, which hides the page body for up to 500ms. Make sure to change the XXXXXXXX for your Site ID. This should be a domain-like value, such as example.com.

<script>
!function(){"use strict";var e=window.VIP_EXP=window.VIP_EXP||{config:{}};e.loadVIPExp=function(t){var n=arguments.length>1&&void 0!==arguments[1]?arguments[1]:{};t&&(e.config=n,e.config.apikey=t,function(e){if(!e)return;var t="https://experiments.parsely.com/vip-experiments.js"+"?apiKey=".concat(e),n=document.createElement("script");n.src=t,n.type="text/javascript",n.fetchPriority="high";var i=document.getElementsByTagName("script")[0];i&&i.parentNode&&i.parentNode.insertBefore(n,i)}(t),n.enableFlickerControl&&function(){var t,n;if(null!==(t=performance)&&void 0!==t&&null!==(n=t.getEntriesByName)&&void 0!==n&&null!==(n=n.call(t,"first-contentful-paint"))&&void 0!==n&&n[0])return;var i="vipexp-fooc-prevention";e.config.disableFlickerControl=function(){var e=document.getElementById(i);null!=e&&e.parentNode&&e.parentNode.removeChild(e)};var o=document.createElement("style");o.setAttribute("type","text/css"),o.appendChild(document.createTextNode("body { visibility: hidden; }")),o.id=i,document.head.appendChild(o),window.setTimeout(e.config.disableFlickerControl,500)}())},

// your Parse.ly Site ID here
e.loadVIPExp("XXXXXXXX",{
enableFlickerControl: true
})
}();
</script>

Installation Through Tag Manager

We do not recommend installing the script using a tag manager. This method may introduce a delay in script execution which can lead to experiments running less often.

Configuration options

By default, headlines are swapped during the initial page load. If your website loads articles dynamically and you want headlines to be updated for newly added anchors, you can enable live updates.

When live updates are enabled, the system watches the page after the initial swaps and automatically updates headlines for eligible anchors as new links appear. You can also configure how long this lasts – by default, we watch the page for 30 seconds.

Options

liveUpdatesEnabled: false 
liveUpdateTimeout: 30000 // 30 seconds in milliseconds

Example of how to enable live updates for 5 seconds using the One-Line Snippet:

<script src="https://experiments.parsely.com/vip-experiments.js?apiKey=XXXXX" data-enable-live-updates="true" data-live-update-timeout="5000"><script>

Example for the advanced integration option:

// Rest of script

e.loadVIPExp("XXXXXXXX",{
enableFlickerControl: true,
liveUpdatesEnabled: true,
liveUpdateTimeout: 5000
})

Content Security Policy (CSP)

If you use script-src, content-src, or img-src directives, update them to allow our domains:

script-src 'self' https://experiments.parsely.com;
img-src 'self' https://tr-experiments.parsely.com;
content-src 'self' https://tr-experiments.parsely.com;

For the advanced installation snippet, ensure your CSP allows script execution. You can achieve this using a nonce, hash, or the ‘unsafe-inline’ directive (though the latter is generally not recommended).

script-src 'self' 'unsafe-inline' https://experiments.parsely.com;

For more details, see MDN: Content Security Policy.

Disable Headline testing

To disable Headline Testing, you can remove the JS Snippet from your website. Once the script is no longer loaded, no tests will be performed. 

The same applies if you would like to disable headline tests on specific pages. Only where the script is loaded tests will be performed and data gathered.

A/B Testing vs Multi-Armed Bandit (MAB)

Parse.ly’s headline testing uses a Multi-Armed Bandit (MAB) algorithm rather than traditional A/B testing. Here’s how the two approaches differ, and why MAB is better suited for headline optimization.

Multi-Armed Bandit algorithms do more than just test headline variants, they optimize traffic distribution and adapt in real time. As the system collects performance data, it dynamically adjusts how traffic is allocated, potentially sending more visitors to the better-performing headlines as soon as there is enough evidence to support the shift. This allows the best headline to reach a larger portion of the audience more quickly, while also ensuring that just enough exposure is given to the weaker variants. Multi-Armed Bandit is especially useful in environments where content has a short lifecycle or where fast optimization is required.

Troubleshooting

Valid HTML Structure vs Invalid HTML Structure

To ensure accurate testing and tracking, Parse.ly’s headline testing requires a specific HTML structure. The system identifies the control headline using an anchor (<a>) tag that must follow a strict format.

Requirements for a Valid HTML Structure

• The control headline must be included within the anchor tag’s content.
• The anchor tag may contain multiple child elements, but at least one child must contain only the control headline text.
• The control headline must not appear outside the anchor.

Valid Example

<a href="/example-article">
  <h2>This is the control headline</h2>
</a>

In this case, the anchor tag contains one child element, and that element holds only the control headline. This structure is valid and supported.

Invalid Examples

Example 1: Additional text

<a href="/example-article">
  <h2>Read more: This is the control headline</h2>
</a>

This is invalid because the tag containing the control headline contains additional text. The system cannot reliably extract the control headline.

Example 2: Headline outside the anchor

<div class="caption">
  <div class="category">Tips</div>
  <h2 class="title">How to Optimize Headlines with Testing</h2>
</div>
<a href="/article/headline-testing-101" title="How to Optimize Headlines with Testing"></a>

This is invalid because the control headline is placed outside the anchor. Although the anchor includes a title attribute, this is not sufficient for headline testing. The system requires the actual headline text to appear inside the anchor’s content.

Headlines Are Not Being Replaced

Possible Issue:
The control headline is being selected as the variant

Solution:
Check if the anchor has the vipexp-swapped-element class. If it does, the system is working as expected.

Possible Issue:
The script is being executed after the first contentful paint.

Solution:
Try loading the script synchronously or enabling flicker control.

Possible Issue:
Mismatch in the URL or control text.

Solution:
Ensure the anchor’s href attribute includes the canonical URL path. Additionally, the anchor must contain either:

1. The exact headline text with no extra text (though it may have child elements), or
2. At least one child element that exclusively contains the exact headline text.

I am unable to see the Create Test , Start Test , or End Test buttons

Possible Issue:
You do not have the necessary permissions to create or end tests.

Solution:
By default, Admins have the permission to manage headline tests. If you are not an Admin, you can request for the Manage Headline Test permission from an Admin, as shown above.

Debug Panel

The Debug Panel is a built-in tool to help you verify that headline experiments are functioning correctly and to provide detailed runtime metrics. It is useful for debugging implementation issues and understanding the performance characteristics of the headline testing feature.

How to Access the Debug Panel

To open the Debug Panel on any page where headline testing is active, append the following query parameter to the page URL:

?vipexp-debug=true

This will display an “Open Debug” button in the top-right corner of the page. Clicking this button will open the Debug Panel, which includes four tabs:

• Experiments
• Navigation
• Paint
• Resources

Experiments Tab

The Experiments tab displays information about headline test execution and runtime behavior.

General Metrics

• Total Run Duration (ms): Time taken to complete processing and headline replacement on the page.
• Total Matching URLs: Number of URLs on the page where the system detected valid experiment targets.
• Total Headlines Swapped: Number of headlines successfully replaced with a test variant.

Note: The number of "Matching URLs" may differ from "Headlines Swapped" if any URL contains no recognizable control headline.

Per Experiment Metrics

For each active experiment on the page, the following metrics are shown:

• Experiment Run Duration (ms): Total runtime from detection to completion of the experiment.
• Control Headline: The original headline used as the control.
• Selected Variant: The variant chosen for display based on optimization.
• Matching URL: The URL where the experiment was detected.
• Headlines Swapped: Whether the replacement was successful.
• Ran Before FCP: Whether the swap occurred before the first contentful paint.
• Headline Swap Duration (ms): Time taken to replace the headline.

Navigation Tab

This tab shows:

• DOM Interactive (ms): Time from page load start until the DOM is ready for interaction.

Paint Tab

This tab provides:

• First Contentful Paint (ms): Time from navigation start until the first piece of content is rendered to the screen.

Resources Tab

This tab shows performance metrics related to loading the experiment bundle:

• Bundle Fetch Start Time (ms): Timestamp for when the fetch request for the experiment JavaScript bundle began.
  
• Bundle Load Duration (ms): Time taken to fully load and parse the bundle.
  

Clear Local Data Button

The Debug Panel includes a Clear Local Data button. This allows you to reset all locally stored experiment data and browser cache for the current user. Using this button will:

• Remove stored variant selections.
  
• Reset all active experiment states.
  

This feature is especially useful when debugging persistent issues or validating variant selection logic.