java - Using JSOUP to Login to ConEd website -


i have read extensively how , have tried number of different variations, can't work.

basically, want login conedison website , scrape billing history. here have:

connection.response loginform = jsoup.connect("https://apps.coned.com/cemyaccount/nonmemberpages/login.aspx?lang=eng")                     .data("_lastfocus","")                     .data("_eventtarget","")                     .data("_eventargument","")                     .data("_viewstate", viewstate)                     .data("_eventvalidation", eventvalidation)                     .data("ctl00$main$login1$username", username)                     .data("ctl00$main$login1$password", password)                     .data("ctl00$main$login1$loginbutton", "sign in")                     .useragent("mozilla/5.0")                     .method(method.post)                     .execute();              map<string, string> logincookies = loginform.cookies();              document document = jsoup.connect("https://apps.coned.com/cemyaccount/csol/billhistory.aspx?lang=eng")                     .cookies(logincookies)                     .get();              elements data = document.select("table.ctl00_main_lvbillhistory_table1");              //checking if found right page             system.out.println("document: " + document);             //checking if found table             system.out.println("data: " + data); 

i know information correct (though don't know if need pass data parameters no values).

i not getting errors, printing out login page (https://apps.coned.com/cemyaccount/nonmemberpages/login.aspx?lang=eng)

any appreciated.

thanks

edit

so, convinced not able internal page, because after post https://apps.coned.com/cemyaccount/nonmemberpages/login.aspx?lang=eng, 3 cookies set, sends requests https://apps.coned.com/cemyaccount/sessiontransfer.aspx?dir=2asp&url=https://apps.coned.com/csol/mainhome.asp?src=dotnet https://apps.coned.com/csol/sessiontransfer.asp?dir=2asp&guid=3c413f48-d2eb-434a-896b-f9c4eb100714&url=https://apps.coned.com/csol/mainhome.asp?src=dotnet&frm= additional cookies before going homepage

does know how can follow these redirects , cookies in end?

here have, cannot cookies post call.

response response = jsoup                         .connect("https://apps.coned.com/cemyaccount/nonmemberpages/login.aspx?lang=eng")                         .method(method.get)                         .execute();                  map<string, string> cookies = response.cookies();                 cookies.put("nsc_dpofe_bqqt-ttm-pme", response.cookie("nsc_dpofe_bqqt-ttm-ofx"));                 system.out.println("response cookies: " + cookies);                  response = jsoup                         .connect("https://apps.coned.com/cemyaccount/nonmemberpages/login.aspx?lang=eng")                         .header("accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8")                         .header("accept-encoding", "gzip, deflate")                         .header("accept-language", "en-us,en;q=0.8")                         .header("connection", "keep-alive")                         .cookies(cookies)                         .header("host", "apps.coned.com")                         .referrer("https://apps.coned.com/cemyaccount/nonmemberpages/login.aspx?lang=eng&login=0")                         .useragent("mozilla/5.0 (windows nt 6.3; wow64) applewebkit/537.36 (khtml, gecko) chrome/43.0.2357.134 safari/537.36")                         .data("_lastfocus", "")                         .data("_eventtarget", "")                         .data("_eventargument", "")                         .data("_viewstate", viewstate)                         .data("_eventvalidation", eventvalidation)                         .data("ctl00$main$login1$username", username)                         .data("ctl00$main$login1$password", password)                         .data("ctl00$main$login1$loginbutton", "sign in")                         .followredirects(false)                         .method(method.post)                         .execute();                   system.out.println("post cookies: " + response.cookies());                 cookies.putall(response.cookies());                 system.out.println("response cookies: " + cookies);                  response = jsoup                         .connect("https://apps.coned.com/cemyaccount/sessiontransfer.aspx?dir=2asp&url=https:"                                 + "//apps.coned.com/csol/mainhome.asp?src=dotnet")                         .cookies(cookies)                         .followredirects(false)                         .method(method.get)                         .execute();                  cookies.putall(response.cookies());                 system.out.println("response cookies: " + cookies);                 string guid = response.header("location");                  response = jsoup                         .connect("https://apps.coned.com/csol/sessiontransfer.asp?dir=2asp&guid="                                 + guid + "&url=https://apps.coned.com/csol/mainhome.asp"                                 + "?src=dotnet&frm=")                         .cookies(cookies)                         .method(method.get)                         .execute();                  cookies.putall(response.cookies());                 system.out.println("response cookies: " + cookies);                  document datapage = jsoup                         .connect("https://apps.coned.com/cemyaccount/csol/billhistory.aspx?lang=eng")                         .cookies(cookies)                         .get();                  system.out.println("data page: " + datapage);                  elements data = datapage.select("table.ctl00_main_lvbillhistory_table1");                  system.out.println("data: " + data); 

in output cookies, except post cookies blank.

  • open development tools (press f12).
  • select network tab option.
  • up , left there round button. if it's not red, click it. record traffic.
  • visit login page. enter credentials , login.
  • check happens in development tools. there table there, showing files received.
  • check type column. search row has value document. select it. open new screen.
  • select headers , scroll down until locate request headers. there find request made browser. find there values send server.
  • search parameters need. take values , hardcode them code. use same user-agent (just in case) , in general try imitate request code.

all steps above chrome browser.


Comments

Popular posts from this blog

python - Healpy: From Data to Healpix map -

c - Bitwise operation with (signed) enum value -

xslt - Unnest parent nodes by child node -