1.5 inital (#234)

* 1.4.12

* Update README.md

* Update package.json

* Update package.json

* 1.5 initial

- Added parallel mode (experimental, likely no Docker supported)
- Added chalk for clearer logging
- Added support for "SearchOnBing" Activities
- Added more configurable options for certain things
- Redone some of the popup/banner clicking for searching (Redo the entire "popup" clicking, so they're more specifically targeted)
- Axios proxy is now optional in the config
- Fingerprint saving is now optional for desktop and mobile

There needs to be many changes for Docker support, including parallel, the new config settings and general testing!

This is still highly experimental, if you use Docker or want something more stable, use the version before this commit!

* Add queries.json to build

* fix(Login): update URL within authorization loop to reflect current page (#210)

* Many changes

- Updated Packages
- Fixed mobile searches erroring out for dashboard  data
- Reworked "bad page" detection
- Catching more errors
- Reworked the search and "close tabs"
- More fixes to the login
- Fixed to paralell and clustering, thanks to @AariaX

* Docker 1.5 preliminary support (#211)

* Basic docker functionality for 1.5

Preliminary docker support for 1.5. Requires headless=true, clusters=1

* Tidy up timezone, add TZ to compose file

Minor changes that should improve timezone handling, and (hopefully) improve scheduling function

* updated readme to simplify and clarify docker instructions

also removed env vars from table

* Fix syntax for cron

* Fix scheduling, add .gitattributes to normalize line endings

fixed line endings caused by Windows in crontab.template and run_daily.sh, which were breaking cron and script execution in the Docker container.

* Removed unnecessary scheduling key from config.json

This key isn't necessary for docker or the base script.

* Basic docker functionality for 1.5

Preliminary docker support for 1.5. Requires headless=true, clusters=1

Tidied up timezone, add TZ to compose file

Minor changes that should improve timezone handling, and (hopefully) improve scheduling function

updated readme to simplify and clarify docker instructions

also removed env vars from table

Fixed syntax for cron

Fixed scheduling, add .gitattributes to normalize line endings

Fixed line endings caused by Windows in crontab.template and run_daily.sh, which were breaking cron and script execution in the Docker container.

Removed unnecessary scheduling key from config.json

This key isn't necessary for docker or the base script.

* Improve scheduling handling, show logs in console

Fixes scheduling when RUN_ON_START=true, and fixes scheduled runs not appearing in docker logs.

* Update compose.yaml

revert service and container name, revert volumes for better generalization, add tips to environment to set scheduling, timezone and whether to run on container start

* Update README.md

proper container name

Co-authored-by: AariaX <196196201+AariaX@users.noreply.github.com>

---------

Co-authored-by: AariaX <196196201+AariaX@users.noreply.github.com>

* Fixes

- Reworked some of the point counting
- Reverted back to the "playwright" package
- Fixed error throw for emailPrefill

* Update config.json

* Add pre-build script

* Update package.json

* Handle 2FA in parallel mode (#219)

* catch error in reloadBadPage (#220)

* Use pre-build and simplify dockerfile (#218)

This uses the new pre-build script included in package.json to handle deps greatly simplifying the dockerfile.

* Small improvements

* Small fixes

- Fixed log spam for "Waiting for authorization"
- Increased wait from 2 to 5 seconds
- Increased search to "safer" values for default

* Experimenting with selectors

Seeing #223 I want to try if this is a good new addition, since for most user this SHOULD work just as good as clicking the entire box.

* More stuff

- Added ability to exclude logs by their function name
- Now caching config settings

* fix: don't retry on 0 (#228)

* Improvements

- Check if searches for mobile are enabled before creating the new page in the browser
- Return message if mobile search data cannot be found
- Added more selectors for coupons

* Improve Popup Dismissal

- Now executes in Parallel
- Respects a timeout of 1 second

---------

Co-authored-by: AariaX <196196201+AariaX@users.noreply.github.com>
Co-authored-by: mgrimace <55518507+mgrimace@users.noreply.github.com>
This commit is contained in:
Netsky
2025-02-15 16:14:47 +01:00
committed by GitHub
parent d1f4364e18
commit 849406c44f
37 changed files with 937 additions and 586 deletions

View File

@@ -1,19 +1,20 @@
import playwright from 'playwright'
import { BrowserContext } from 'playwright'
import playwright, { BrowserContext } from 'rebrowser-playwright'
import { newInjectedContext } from 'fingerprint-injector'
import { FingerprintGenerator } from 'fingerprint-generator'
import { MicrosoftRewardsBot } from '../index'
import { loadSessionData, saveFingerprintData } from '../util/Load'
import { updateFingerprintUserAgent } from '../util/UserAgent'
import { AccountProxy } from '../interface/Account'
/* Test Stuff
https://abrahamjuliot.github.io/creepjs/
https://botcheck.luminati.io/
http://f.vision/
https://fv.pro/
https://pixelscan.net/
https://www.browserscan.net/
*/
class Browser {
@@ -40,33 +41,35 @@ class Browser {
const sessionData = await loadSessionData(this.bot.config.sessionPath, email, this.bot.isMobile, this.bot.config.saveFingerprint)
const fingerpint = sessionData.fingerprint ? sessionData.fingerprint : this.generateFingerprint()
const fingerprint = sessionData.fingerprint ? sessionData.fingerprint : await this.generateFingerprint()
const context = await newInjectedContext(browser, { fingerprint: fingerpint })
const context = await newInjectedContext(browser as any, { fingerprint: fingerprint })
// Set timeout to preferred amount
context.setDefaultTimeout(this.bot.config?.globalTimeout ?? 30_000)
context.setDefaultTimeout(this.bot.utils.stringToMs(this.bot.config?.globalTimeout ?? 30_000))
await context.addCookies(sessionData.cookies)
if (this.bot.config.saveFingerprint) {
await saveFingerprintData(this.bot.config.sessionPath, email, this.bot.isMobile, fingerpint)
await saveFingerprintData(this.bot.config.sessionPath, email, this.bot.isMobile, fingerprint)
}
this.bot.log('BROWSER', `Created browser with User-Agent: "${fingerpint.fingerprint.navigator.userAgent}"`)
this.bot.log(this.bot.isMobile, 'BROWSER', `Created browser with User-Agent: "${fingerprint.fingerprint.navigator.userAgent}"`)
return context
return context as BrowserContext
}
generateFingerprint() {
async generateFingerprint() {
const fingerPrintData = new FingerprintGenerator().getFingerprint({
devices: this.bot.isMobile ? ['mobile'] : ['desktop'],
operatingSystems: this.bot.isMobile ? ['android'] : ['windows'],
browserListQuery: 'last 2 edge version'
browsers: [{ name: 'edge' }]
})
return fingerPrintData
const updatedFingerPrintData = await updateFingerprintUserAgent(fingerPrintData, this.bot.isMobile)
return updatedFingerPrintData
}
}
export default Browser
export default Browser

View File

@@ -1,12 +1,14 @@
import { Page } from 'playwright'
import { BrowserContext, Page } from 'rebrowser-playwright'
import { CheerioAPI, load } from 'cheerio'
import axios, { AxiosRequestConfig } from 'axios'
import { AxiosRequestConfig } from 'axios'
import { MicrosoftRewardsBot } from '../index'
import { saveSessionData } from '../util/Load'
import { Counters, DashboardData, MorePromotion, PromotionalItem } from './../interface/DashboardData'
import { QuizData } from './../interface/QuizData'
import { AppUserData } from '../interface/AppUserData'
import { EarnablePoints } from '../interface/Points'
export default class BrowserFunc {
@@ -36,19 +38,19 @@ export default class BrowserFunc {
for (let iteration = 1; iteration <= maxIterations; iteration++) {
await this.bot.utils.wait(3000)
await this.bot.browser.utils.tryDismissCookieBanner(page)
await this.bot.browser.utils.tryDismissAllMessages(page)
// Check if account is suspended
const isSuspended = await page.waitForSelector('#suspendedAccountHeader', { state: 'visible', timeout: 2000 }).then(() => true).catch(() => false)
if (isSuspended) {
this.bot.log('GO-HOME', 'This account is suspended!', 'error')
this.bot.log(this.bot.isMobile, 'GO-HOME', 'This account is suspended!', 'error')
throw new Error('Account has been suspended!')
}
try {
// If activities are found, exit the loop
await page.waitForSelector('#more-activities', { timeout: 1000 })
this.bot.log('GO-HOME', 'Visited homepage successfully')
this.bot.log(this.bot.isMobile, 'GO-HOME', 'Visited homepage successfully')
break
} catch (error) {
@@ -64,7 +66,7 @@ export default class BrowserFunc {
await this.bot.utils.wait(2000)
await page.goto(this.bot.config.baseURL)
} else {
this.bot.log('GO-HOME', 'Visited homepage successfully')
this.bot.log(this.bot.isMobile, 'GO-HOME', 'Visited homepage successfully')
break
}
@@ -72,7 +74,7 @@ export default class BrowserFunc {
}
} catch (error) {
throw this.bot.log('GO-HOME', 'An error occurred:' + error, 'error')
throw this.bot.log(this.bot.isMobile, 'GO-HOME', 'An error occurred:' + error, 'error')
}
}
@@ -84,42 +86,49 @@ export default class BrowserFunc {
const dashboardURL = new URL(this.bot.config.baseURL)
const currentURL = new URL(this.bot.homePage.url())
// Should never happen since tasks are opened in a new tab!
if (currentURL.hostname !== dashboardURL.hostname) {
this.bot.log('DASHBOARD-DATA', 'Provided page did not equal dashboard page, redirecting to dashboard page')
await this.goHome(this.bot.homePage)
}
// Reload the page to get new data
await this.bot.homePage.reload({ waitUntil: 'domcontentloaded' })
const scriptContent = await this.bot.homePage.evaluate(() => {
const scripts = Array.from(document.querySelectorAll('script'))
const targetScript = scripts.find(script => script.innerText.includes('var dashboard'))
return targetScript?.innerText ? targetScript.innerText : null
})
if (!scriptContent) {
throw this.bot.log('GET-DASHBOARD-DATA', 'Dashboard data not found within script', 'error')
}
// Extract the dashboard object from the script content
const dashboardData = await this.bot.homePage.evaluate(scriptContent => {
// Extract the dashboard object using regex
const regex = /var dashboard = (\{.*?\});/s
const match = regex.exec(scriptContent)
if (match && match[1]) {
return JSON.parse(match[1])
try {
// Should never happen since tasks are opened in a new tab!
if (currentURL.hostname !== dashboardURL.hostname) {
this.bot.log(this.bot.isMobile, 'DASHBOARD-DATA', 'Provided page did not equal dashboard page, redirecting to dashboard page')
await this.goHome(this.bot.homePage)
}
}, scriptContent)
if (!dashboardData) {
throw this.bot.log('GET-DASHBOARD-DATA', 'Unable to parse dashboard script', 'error')
// Reload the page to get new data
await this.bot.homePage.reload({ waitUntil: 'domcontentloaded' })
const scriptContent = await this.bot.homePage.evaluate(() => {
const scripts = Array.from(document.querySelectorAll('script'))
const targetScript = scripts.find(script => script.innerText.includes('var dashboard'))
return targetScript?.innerText ? targetScript.innerText : null
})
if (!scriptContent) {
throw this.bot.log(this.bot.isMobile, 'GET-DASHBOARD-DATA', 'Dashboard data not found within script', 'error')
}
// Extract the dashboard object from the script content
const dashboardData = await this.bot.homePage.evaluate(scriptContent => {
// Extract the dashboard object using regex
const regex = /var dashboard = (\{.*?\});/s
const match = regex.exec(scriptContent)
if (match && match[1]) {
return JSON.parse(match[1])
}
}, scriptContent)
if (!dashboardData) {
throw this.bot.log(this.bot.isMobile, 'GET-DASHBOARD-DATA', 'Unable to parse dashboard script', 'error')
}
return dashboardData
} catch (error) {
throw this.bot.log(this.bot.isMobile, 'GET-DASHBOARD-DATA', `Error fetching dashboard data: ${error}`, 'error')
}
return dashboardData
}
/**
@@ -136,39 +145,49 @@ export default class BrowserFunc {
* Get total earnable points with web browser
* @returns {number} Total earnable points
*/
async getBrowserEarnablePoints(): Promise<number> {
async getBrowserEarnablePoints(): Promise<EarnablePoints> {
try {
const data = await this.getDashboardData()
let desktopSearchPoints = 0
let mobileSearchPoints = 0
let dailySetPoints = 0
let morePromotionsPoints = 0
// These only include the points from tasks that the script can complete!
let totalEarnablePoints = 0
const data = await this.getDashboardData()
// Desktop Search Points
if (data.userStatus.counters.pcSearch?.length) {
data.userStatus.counters.pcSearch.forEach(x => totalEarnablePoints += (x.pointProgressMax - x.pointProgress))
data.userStatus.counters.pcSearch.forEach(x => desktopSearchPoints += (x.pointProgressMax - x.pointProgress))
}
// Mobile Search Points
if (data.userStatus.counters.mobileSearch?.length) {
data.userStatus.counters.mobileSearch.forEach(x => totalEarnablePoints += (x.pointProgressMax - x.pointProgress))
data.userStatus.counters.mobileSearch.forEach(x => mobileSearchPoints += (x.pointProgressMax - x.pointProgress))
}
// Daily Set
data.dailySetPromotions[this.bot.utils.getFormattedDate()]?.forEach(x => totalEarnablePoints += (x.pointProgressMax - x.pointProgress))
data.dailySetPromotions[this.bot.utils.getFormattedDate()]?.forEach(x => dailySetPoints += (x.pointProgressMax - x.pointProgress))
// More Promotions
if (data.morePromotions?.length) {
data.morePromotions.forEach(x => {
// Only count points from supported activities
if (['quiz', 'urlreward'].includes(x.promotionType) && x.exclusiveLockedFeatureStatus !== 'locked') {
totalEarnablePoints += (x.pointProgressMax - x.pointProgress)
morePromotionsPoints += (x.pointProgressMax - x.pointProgress)
}
})
}
return totalEarnablePoints
const totalEarnablePoints = desktopSearchPoints + mobileSearchPoints + dailySetPoints + morePromotionsPoints
return {
dailySetPoints,
morePromotionsPoints,
desktopSearchPoints,
mobileSearchPoints,
totalEarnablePoints
}
} catch (error) {
throw this.bot.log('GET-BROWSER-EARNABLE-POINTS', 'An error occurred:' + error, 'error')
throw this.bot.log(this.bot.isMobile, 'GET-BROWSER-EARNABLE-POINTS', 'An error occurred:' + error, 'error')
}
}
@@ -176,13 +195,18 @@ export default class BrowserFunc {
* Get total earnable points with mobile app
* @returns {number} Total earnable points
*/
async getAppEarnablePoints(accessToken: string): Promise<number> {
async getAppEarnablePoints(accessToken: string) {
try {
const points = {
readToEarn: 0,
checkIn: 0,
totalEarnablePoints: 0
}
const eligibleOffers = [
'ENUS_readarticle3_30points',
'Gamification_Sapphire_DailyCheckIn'
]
let totalEarnablePoints = 0
const data = await this.getDashboardData()
let geoLocale = data.userProfile.attributes.country
@@ -198,27 +222,29 @@ export default class BrowserFunc {
}
}
const userDataResponse: AppUserData = (await axios(userDataRequest)).data
const userDataResponse: AppUserData = (await this.bot.axios.request(userDataRequest)).data
const userData = userDataResponse.response
const eligibleActivities = userData.promotions.filter((x) => eligibleOffers.includes(x.attributes.offerid ?? ''))
for (const item of eligibleActivities) {
if (item.attributes.type === 'msnreadearn') {
totalEarnablePoints += parseInt(item.attributes.pointmax ?? '') - parseInt(item.attributes.pointprogress ?? '')
points.readToEarn = parseInt(item.attributes.pointmax ?? '') - parseInt(item.attributes.pointprogress ?? '')
break
} else if (item.attributes.type === 'checkin') {
const checkInDay = parseInt(item.attributes.progress ?? '') % 7
if (checkInDay < 6 && (new Date()).getDate() != (new Date(item.attributes.last_updated ?? '')).getDate()) {
totalEarnablePoints += parseInt(item.attributes['day_' + (checkInDay + 1) + '_points'] ?? '')
points.checkIn = parseInt(item.attributes['day_' + (checkInDay + 1) + '_points'] ?? '')
}
break
}
}
return totalEarnablePoints
points.totalEarnablePoints = points.readToEarn + points.checkIn
return points
} catch (error) {
throw this.bot.log('GET-APP-EARNABLE-POINTS', 'An error occurred:' + error, 'error')
throw this.bot.log(this.bot.isMobile, 'GET-APP-EARNABLE-POINTS', 'An error occurred:' + error, 'error')
}
}
@@ -232,7 +258,7 @@ export default class BrowserFunc {
return data.userStatus.availablePoints
} catch (error) {
throw this.bot.log('GET-CURRENT-POINTS', 'An error occurred:' + error, 'error')
throw this.bot.log(this.bot.isMobile, 'GET-CURRENT-POINTS', 'An error occurred:' + error, 'error')
}
}
@@ -258,14 +284,14 @@ export default class BrowserFunc {
const quizData = JSON.parse(match[1])
return quizData
} else {
throw this.bot.log('GET-QUIZ-DATA', 'Quiz data not found within script', 'error')
throw this.bot.log(this.bot.isMobile, 'GET-QUIZ-DATA', 'Quiz data not found within script', 'error')
}
} else {
throw this.bot.log('GET-QUIZ-DATA', 'Script containing quiz data not found', 'error')
throw this.bot.log(this.bot.isMobile, 'GET-QUIZ-DATA', 'Script containing quiz data not found', 'error')
}
} catch (error) {
throw this.bot.log('GET-QUIZ-DATA', 'An error occurred:' + error, 'error')
throw this.bot.log(this.bot.isMobile, 'GET-QUIZ-DATA', 'An error occurred:' + error, 'error')
}
}
@@ -277,7 +303,7 @@ export default class BrowserFunc {
return true
} catch (error) {
this.bot.log('QUIZ-REFRESH', 'An error occurred:' + error, 'error')
this.bot.log(this.bot.isMobile, 'QUIZ-REFRESH', 'An error occurred:' + error, 'error')
return false
}
}
@@ -293,7 +319,7 @@ export default class BrowserFunc {
}
}
async refreshCheerio(page: Page): Promise<CheerioAPI> {
async loadInCheerio(page: Page): Promise<CheerioAPI> {
const html = await page.content()
const $ = load(html)
@@ -311,10 +337,24 @@ export default class BrowserFunc {
selector = `a[href*="${element.attribs.href}"]`
}
} catch (error) {
this.bot.log('GET-PUNCHCARD-ACTIVITY', 'An error occurred:' + error, 'error')
this.bot.log(this.bot.isMobile, 'GET-PUNCHCARD-ACTIVITY', 'An error occurred:' + error, 'error')
}
return selector
}
async closeBrowser(browser: BrowserContext, email: string) {
try {
// Save cookies
await saveSessionData(this.bot.config.sessionPath, browser, email, this.bot.isMobile)
await this.bot.utils.wait(2000)
// Close browser
await browser.close()
this.bot.log(this.bot.isMobile, 'CLOSE-BROWSER', 'Browser closed cleanly!')
} catch (error) {
throw this.bot.log(this.bot.isMobile, 'CLOSE-BROWSER', 'An error occurred:' + error, 'error')
}
}
}

View File

@@ -1,4 +1,5 @@
import { Page } from 'playwright'
import { Page } from 'rebrowser-playwright'
import { load } from 'cheerio'
import { MicrosoftRewardsBot } from '../index'
@@ -13,66 +14,41 @@ export default class BrowserUtil {
async tryDismissAllMessages(page: Page): Promise<boolean> {
const buttons = [
{ selector: '#acceptButton', label: 'AcceptButton' },
{ selector: '.ext-secondary.ext-button', label: '"Skip for now" Button' },
{ selector: '#iLandingViewAction', label: 'iLandingViewAction' },
{ selector: '#iShowSkip', label: 'iShowSkip' },
{ selector: '#iNext', label: 'iNext' },
{ selector: '#iLooksGood', label: 'iLooksGood' },
{ selector: '#idSIButton9', label: 'idSIButton9' },
{ selector: '.ms-Button.ms-Button--primary', label: 'Primary Button' }
{ selector: '.ms-Button.ms-Button--primary', label: 'Primary Button' },
{ selector: '.c-glyph.glyph-cancel', label: 'Mobile Welcome Button' },
{ selector: '.maybe-later', label: 'Mobile Rewards App Banner' },
{ selector: '//div[@id="cookieConsentContainer"]//button[contains(text(), "Accept")]', label: 'Accept Cookie Consent Container' },
{ selector: '#bnp_btn_accept', label: 'Bing Cookie Banner' },
{ selector: '#reward_pivot_earn', label: 'Reward Coupon Accept' }
]
let result = false
for (const button of buttons) {
const dismissTasks = buttons.map(async (button) => {
try {
const element = await page.waitForSelector(button.selector, { state: 'visible', timeout: 1000 })
if (element) {
await element.click()
result = true
const element = page.locator(button.selector)
if (await element.first().isVisible({ timeout: 1000 })) {
await element.first().click({ timeout: 1000 })
this.bot.log(this.bot.isMobile, 'DISMISS-ALL-MESSAGES', `Dismissed: ${button.label}`)
return true
}
} catch (error) {
continue
// Ignore errors and continue
}
}
return false
})
return result
}
async tryDismissCookieBanner(page: Page): Promise<void> {
try {
await page.waitForSelector('#cookieConsentContainer', { timeout: 1000 })
const cookieBanner = await page.$('#cookieConsentContainer')
if (cookieBanner) {
const button = await cookieBanner.$('button')
if (button) {
await button.click()
await this.bot.utils.wait(2000)
}
}
} catch (error) {
// Continue if element is not found or other error occurs
}
}
async tryDismissBingCookieBanner(page: Page): Promise<void> {
try {
await page.waitForSelector('#bnp_btn_accept', { timeout: 1000 })
const cookieBanner = await page.$('#bnp_btn_accept')
if (cookieBanner) {
await cookieBanner.click()
}
} catch (error) {
// Continue if element is not found or other error occurs
}
const results = await Promise.allSettled(dismissTasks)
return results.some(result => result.status === 'fulfilled' && result.value === true)
}
async getLatestTab(page: Page): Promise<Page> {
try {
await this.bot.utils.wait(500)
await this.bot.utils.wait(1000)
const browser = page.context()
const pages = browser.pages()
@@ -82,9 +58,9 @@ export default class BrowserUtil {
return newTab
}
throw this.bot.log('GET-NEW-TAB', 'Unable to get latest tab', 'error')
throw this.bot.log(this.bot.isMobile, 'GET-NEW-TAB', 'Unable to get latest tab', 'error')
} catch (error) {
throw this.bot.log('GET-NEW-TAB', 'An error occurred:' + error, 'error')
throw this.bot.log(this.bot.isMobile, 'GET-NEW-TAB', 'An error occurred:' + error, 'error')
}
}
@@ -97,19 +73,19 @@ export default class BrowserUtil {
let homeTabURL: URL
if (!homeTab) {
throw this.bot.log('GET-TABS', 'Home tab could not be found!', 'error')
throw this.bot.log(this.bot.isMobile, 'GET-TABS', 'Home tab could not be found!', 'error')
} else {
homeTabURL = new URL(homeTab.url())
if (homeTabURL.hostname !== 'rewards.bing.com') {
throw this.bot.log('GET-TABS', 'Reward page hostname is invalid: ' + homeTabURL.host, 'error')
throw this.bot.log(this.bot.isMobile, 'GET-TABS', 'Reward page hostname is invalid: ' + homeTabURL.host, 'error')
}
}
const workerTab = pages[2]
if (!workerTab) {
throw this.bot.log('GET-TABS', 'Worker tab could not be found!', 'error')
throw this.bot.log(this.bot.isMobile, 'GET-TABS', 'Worker tab could not be found!', 'error')
}
return {
@@ -118,7 +94,24 @@ export default class BrowserUtil {
}
} catch (error) {
throw this.bot.log('GET-TABS', 'An error occurred:' + error, 'error')
throw this.bot.log(this.bot.isMobile, 'GET-TABS', 'An error occurred:' + error, 'error')
}
}
async reloadBadPage(page: Page): Promise<void> {
try {
const html = await page.content().catch(() => '')
const $ = load(html)
const isNetworkError = $('body.neterror').length
if (isNetworkError) {
this.bot.log(this.bot.isMobile, 'RELOAD-BAD-PAGE', 'Bad page detected, reloading!')
await page.reload()
}
} catch (error) {
throw this.bot.log(this.bot.isMobile, 'RELOAD-BAD-PAGE', 'An error occurred:' + error, 'error')
}
}