java - Using JSOUP to Login to ConEd website -
i have read extensively how , have tried number of different variations, can't work.
basically, want login conedison website , scrape billing history. here have:
connection.response loginform = jsoup.connect("https://apps.coned.com/cemyaccount/nonmemberpages/login.aspx?lang=eng") .data("_lastfocus","") .data("_eventtarget","") .data("_eventargument","") .data("_viewstate", viewstate) .data("_eventvalidation", eventvalidation) .data("ctl00$main$login1$username", username) .data("ctl00$main$login1$password", password) .data("ctl00$main$login1$loginbutton", "sign in") .useragent("mozilla/5.0") .method(method.post) .execute(); map<string, string> logincookies = loginform.cookies(); document document = jsoup.connect("https://apps.coned.com/cemyaccount/csol/billhistory.aspx?lang=eng") .cookies(logincookies) .get(); elements data = document.select("table.ctl00_main_lvbillhistory_table1"); //checking if found right page system.out.println("document: " + document); //checking if found table system.out.println("data: " + data);
i know information correct (though don't know if need pass data parameters no values).
i not getting errors, printing out login page (https://apps.coned.com/cemyaccount/nonmemberpages/login.aspx?lang=eng)
any appreciated.
thanks
edit
so, convinced not able internal page, because after post https://apps.coned.com/cemyaccount/nonmemberpages/login.aspx?lang=eng, 3 cookies set, sends requests https://apps.coned.com/cemyaccount/sessiontransfer.aspx?dir=2asp&url=https://apps.coned.com/csol/mainhome.asp?src=dotnet https://apps.coned.com/csol/sessiontransfer.asp?dir=2asp&guid=3c413f48-d2eb-434a-896b-f9c4eb100714&url=https://apps.coned.com/csol/mainhome.asp?src=dotnet&frm= additional cookies before going homepage
does know how can follow these redirects , cookies in end?
here have, cannot cookies post call.
response response = jsoup .connect("https://apps.coned.com/cemyaccount/nonmemberpages/login.aspx?lang=eng") .method(method.get) .execute(); map<string, string> cookies = response.cookies(); cookies.put("nsc_dpofe_bqqt-ttm-pme", response.cookie("nsc_dpofe_bqqt-ttm-ofx")); system.out.println("response cookies: " + cookies); response = jsoup .connect("https://apps.coned.com/cemyaccount/nonmemberpages/login.aspx?lang=eng") .header("accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8") .header("accept-encoding", "gzip, deflate") .header("accept-language", "en-us,en;q=0.8") .header("connection", "keep-alive") .cookies(cookies) .header("host", "apps.coned.com") .referrer("https://apps.coned.com/cemyaccount/nonmemberpages/login.aspx?lang=eng&login=0") .useragent("mozilla/5.0 (windows nt 6.3; wow64) applewebkit/537.36 (khtml, gecko) chrome/43.0.2357.134 safari/537.36") .data("_lastfocus", "") .data("_eventtarget", "") .data("_eventargument", "") .data("_viewstate", viewstate) .data("_eventvalidation", eventvalidation) .data("ctl00$main$login1$username", username) .data("ctl00$main$login1$password", password) .data("ctl00$main$login1$loginbutton", "sign in") .followredirects(false) .method(method.post) .execute(); system.out.println("post cookies: " + response.cookies()); cookies.putall(response.cookies()); system.out.println("response cookies: " + cookies); response = jsoup .connect("https://apps.coned.com/cemyaccount/sessiontransfer.aspx?dir=2asp&url=https:" + "//apps.coned.com/csol/mainhome.asp?src=dotnet") .cookies(cookies) .followredirects(false) .method(method.get) .execute(); cookies.putall(response.cookies()); system.out.println("response cookies: " + cookies); string guid = response.header("location"); response = jsoup .connect("https://apps.coned.com/csol/sessiontransfer.asp?dir=2asp&guid=" + guid + "&url=https://apps.coned.com/csol/mainhome.asp" + "?src=dotnet&frm=") .cookies(cookies) .method(method.get) .execute(); cookies.putall(response.cookies()); system.out.println("response cookies: " + cookies); document datapage = jsoup .connect("https://apps.coned.com/cemyaccount/csol/billhistory.aspx?lang=eng") .cookies(cookies) .get(); system.out.println("data page: " + datapage); elements data = datapage.select("table.ctl00_main_lvbillhistory_table1"); system.out.println("data: " + data);
in output cookies, except post cookies blank.
- open development tools (press f12).
- select
network
tab option. - up , left there round button. if it's not red, click it. record traffic.
- visit login page. enter credentials , login.
- check happens in development tools. there table there, showing files received.
- check
type
column. search row has valuedocument
. select it. open new screen. - select
headers
, scroll down until locaterequest headers
. there find request made browser. find there values send server. - search parameters need. take values , hardcode them code. use same user-agent (just in case) , in general try imitate request code.
all steps above chrome browser.
Comments
Post a Comment