Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
16 out of 60 is failed (threshold=3) [.............X......XX.XX..X......XXX....XX.....XX...X.XX...] duration:
39.540515651s
FAIL (2025-07-09 01:58:10 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 39.540539016s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [................X......................X.................X..] duration:
20.079341315s
FAIL (2025-07-09 01:58:05 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
em: *pb.BatchRequestItem_SegmentCommit
Jul 9 01:55:26.378238935 (376h45m1.14888026s) [upload=301404]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 01:55:26.681901575 (376h45m1.45254291s) [upload=301404]: Done uploading segments...
Jul 9 01:55:26.681918448 (376h45m1.452559783s) [upload=301404]: Upload finished. err: <nil>
Jul 9 01:56:07.557068843 (376h45m42.327710178s) [upload=301419]: Starting upload
Jul 9 01:56:07.557123908 (376h45m42.327765243s) [upload=301419]: Done starting upload
Jul 9 01:56:07.557294845 (376h45m42.32793618s) [upload=301419]: Uploading segments...
Jul 9 01:56:07.557334942 (376h45m42.327976267s) [upload=301419]: Got next segment. Inline: false
Jul 9 01:56:07.557358848 (376h45m42.328000183s) [upload=301419]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 01:56:07.557363737 (376h45m42.328005072s) [upload=301419]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 01:56:07.786768822 (376h45m42.557410157s) [seg_pos={0 0}, upload=301419]: Begin upload segment...
Jul 9 01:56:07.786807697 (376h45m42.557449032s) [seg_pos={0 0}, upload=301419]: Done begin upload segment.
Jul 9 01:56:07.786827744 (376h45m42.557469069s) [noise=true , address=45.91.116.13:4154 , piece_id=WT7WL2P7RMJFA7XIZ3KNRVJ2RWDQGQBCM3J65X2X3MQ6WSAKHOIQ, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.786867240 (376h45m42.557508576s) [noise=true , address=104.36.82.2:4732 , piece_id=HF35AWQTHK4B5XQXLUOM5HPEBINSUIXGA3AAMZFEHZW4GDRDT4KQ, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.787161643 (376h45m42.557802968s) [noise=true , address=104.36.82.16:4294 , piece_id=TWDA4QZ5WAQF3OOXDDNAHBKYM5OAKBZ34QWW2FYXHJ6KGKDTDFRQ, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.787243289 (376h45m42.557884614s) [noise=true , address=77.90.31.11:4555 , piece_id=A7UGOPRF44445XR4OBEODVMMJU4HED7JUOPORHJKXVEPJO6KYIWQ, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.787405490 (376h45m42.558046815s) [noise=true , address=77.90.31.10:4518 , piece_id=ZC7HW4EDPE6UU2MLOHGHMYKBA4NXTJEL4I6CP6FK62QIAOEUT4ZQ, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.787501334 (376h45m42.558142669s) [noise=true , address=45.91.116.6:4975 , piece_id=MRETRUDTUOXXPHG4P25RMAJP4EDX4SS4B7O22PPPIQBSULZ52KNA, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.787663624 (376h45m42.558304959s) [noise=true , address=104.36.82.4:4801 , piece_id=OVVFQSK4TYRAXFNR6NZ74463CDC5BPKVROPU45ATGOTCZDUFDWHQ, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.787980670 (376h45m42.558622005s) [noise=true , address=45.91.116.11:4117 , piece_id=VDNXPWCQBXOOEAEJB7YB75AWP25A2AMCLTOT6KHFETFUXZ6UH3IQ, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.788245387 (376h45m42.558886712s) [noise=true , address=104.36.82.10:4078 , piece_id=ORPJSLOMY4IAB2NWYOQ7RCZCLEIQ25CIXFKKTWW5LBO5BFPPY64Q, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.788473754 (376h45m42.559115079s) [noise=true , address=45.91.116.7:4009 , piece_id=EEGGWRYW7EDJOVZDWO36KAGPFBQFSK5FJ7YU2FPBBWSIGV2KM4UA, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.788656052 (376h45m42.559297387s) [noise=true , address=104.36.82.6:4876 , piece_id=TUZF2S3DJSUK6MMNEQYWR35I56NAFLDRFPL646TIAGIUNVWE2ZGA, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.788848981 (376h45m42.559490316s) [noise=true , address=45.91.116.10:4081 , piece_id=4ZKRZFKWSILUJ3QNHIDRBGUETYMO7UZI5G5XR64XI2ANHXRCKAZQ, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.789032262 (376h45m42.559673597s) [noise=true , address=104.36.82.9:4006 , piece_id=P7YEMWPREVLDDFE7JDYAOEZJIYCAL3OAKKJJDC6EBMLLYSVRL73Q, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.789272181 (376h45m42.559913506s) [noise=true , address=77.90.31.19:4843 , piece_id=5FRTA2EJNZVNHV47EWYRFUH6VW2FJN2EBAKBRJWMP66BXF6SZF6A, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.789459429 (376h45m42.560100764s) [noise=true , address=45.91.116.8:4047 , piece_id=VISW3TARCTQMHB3662HBA2I7UW4IIREOJ32XOSBNTRPPTB4KD77A, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.789640225 (376h45m42.56028155s) [noise=true , address=77.90.31.21:4877 , piece_id=EN7QS3CEXXIBY7FU6Y32LLDKYCSVRJQ4G7RXRE2DV7CUPMZBVHDQ, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:09.422697440 (376h45m44.193338775s) [upload=301419]: Next returned nil segment
Jul 9 01:56:09.422714812 (376h45m44.193356148s) [upload=301419]: Waiting for error group managing segments...
Jul 9 01:56:09.703007945 (376h45m44.47364928s) [noise=true , address=104.36.82.9:4006 , piece_id=P7YEMWPREVLDDFE7JDYAOEZJIYCAL3OAKKJJDC6EBMLLYSVRL73Q, seg_pos={0 0}, upload=301419]: Done uploading piece. err: <nil>
Jul 9 01:56:09.780867601 (376h45m44.551508936s) [noise=true , address=104.36.82.16:4294 , piece_id=TWDA4QZ5WAQF3OOXDDNAHBKYM5OAKBZ34QWW2FYXHJ6KGKDTDFRQ, seg_pos={0 0}, upload=301419]: Done uploading piece. err: <nil>
Jul 9 01:56:09.840231843 (376h45m44.610873178s) [noise=true , address=45.91.116.6:4975 , piece_id=MRETRUDTUOXXPHG4P25RMAJP4EDX4SS4B7O22PPPIQBSULZ52KNA, seg_pos={0 0}, upload=301419]: Done uploading piece. err: <nil>
Jul 9 01:56:09.841984998 (376h45m44.612626363s) [noise=true , address=45.91.116.10:4081 , piece_id=4ZKRZFKWSILUJ3QNHIDRBGUETYMO7UZI5G5XR64XI2ANHXRCKAZQ, seg_pos={0 0}, upload=301419]: Done uploading piece. err: <nil>
Jul 9 01:56:09.850995516 (376h45m44.621636851s) [noise=true , address=45.91.116.13:4154 , piece_id=WT7WL2P7RMJFA7XIZ3KNRVJ2RWDQGQBCM3J65X2X3MQ6WSAKHOIQ, seg_pos={0 0}, upload=301419]: Done uploading piece. err: <nil>
Jul 9 01:56:09.860567239 (376h45m44.631208574s) [noise=true , address=45.91.116.7:4009 , piece_id=EEGGWRYW7EDJOVZDWO36KAGPFBQFSK5FJ7YU2FPBBWSIGV2KM4UA, seg_pos={0 0}, upload=301419]: Done uploading piece. err: <nil>
Jul 9 01:56:09.871129736 (376h45m44.641771071s) [noise=true , address=77.90.31.19:4843 , piece_id=5FRTA2EJNZVNHV47EWYRFUH6VW2FJN2EBAKBRJWMP66BXF6SZF6A, seg_pos={0 0}, upload=301419]: Done uploading piece. err: <nil>
Jul 9 01:56:09.874489343 (376h45m44.645130678s) [noise=true , address=77.90.31.11:4555 , piece_id=A7UGOPRF44445XR4OBEODVMMJU4HED7JUOPORHJKXVEPJO6KYIWQ, seg_pos={0 0}, upload=301419]: Done uploading piece. err: <nil>
Jul 9 01:56:09.875282992 (376h45m44.645924327s) [noise=true , address=77.90.31.21:4877 , piece_id=EN7QS3CEXXIBY7FU6Y32LLDKYCSVRJQ4G7RXRE2DV7CUPMZBVHDQ, seg_pos={0 0}, upload=301419]: Done uploading piece. err: <nil>
Jul 9 01:56:09.882277543 (376h45m44.652918878s) [noise=true , address=77.90.31.10:4518 , piece_id=ZC7HW4EDPE6UU2MLOHGHMYKBA4NXTJEL4I6CP6FK62QIAOEUT4ZQ, seg_pos={0 0}, upload=301419]: Done uploading piece. err: <nil>
Jul 9 01:56:09.909865152 (376h45m44.680506487s) [noise=true , address=45.91.116.11:4117 , piece_id=VDNXPWCQBXOOEAEJB7YB75AWP25A2AMCLTOT6KHFETFUXZ6UH3IQ, seg_pos={0 0}, upload=301419]: Done uploading piece. err: <nil>
Jul 9 01:56:09.930049268 (376h45m44.700690603s) [noise=true , address=45.91.116.8:4047 , piece_id=VISW3TARCTQMHB3662HBA2I7UW4IIREOJ32XOSBNTRPPTB4KD77A, seg_pos={0 0}, upload=301419]: Done uploading piece. err: <nil>
Jul 9 01:56:36.916221275 (376h46m11.68686263s) [noise=true , address=104.36.82.6:4876 , piece_id=TUZF2S3DJSUK6MMNEQYWR35I56NAFLDRFPL646TIAGIUNVWE2ZGA, seg_pos={0 0}, upload=301419]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 188.166.225.170:36068->104.36.82.6:4876: use of closed network connection; write tcp 188.166.225.170:36068->104.36.82.6:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36068->104.36.82.6:4876: use of closed network connection
Jul 9 01:56:36.916318731 (376h46m11.686960076s) [noise=true , address=104.36.82.4:4801 , piece_id=OVVFQSK4TYRAXFNR6NZ74463CDC5BPKVROPU45ATGOTCZDUFDWHQ, seg_pos={0 0}, upload=301419]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 188.166.225.170:36588->104.36.82.4:4801: use of closed network connection; context deadline exceeded
Jul 9 01:56:36.916361463 (376h46m11.687002818s) [noise=true , address=104.36.82.2:4732 , piece_id=HF35AWQTHK4B5XQXLUOM5HPEBINSUIXGA3AAMZFEHZW4GDRDT4KQ, seg_pos={0 0}, upload=301419]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: write tcp 188.166.225.170:44126->104.36.82.2:4732: use of closed network connection; write tcp 188.166.225.170:44126->104.36.82.2:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44126->104.36.82.2:4732: use of closed network connection
Jul 9 01:56:36.916428551 (376h46m11.687069916s) [noise=true , address=104.36.82.10:4078 , piece_id=ORPJSLOMY4IAB2NWYOQ7RCZCLEIQ25CIXFKKTWW5LBO5BFPPY64Q, seg_pos={0 0}, upload=301419]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 188.166.225.170:53862->104.36.82.10:4078: use of closed network connection; write tcp 188.166.225.170:53862->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53862->104.36.82.10:4078: use of closed network connection
Jul 9 01:56:36.916470932 (376h46m11.687112338s) [seg_pos={0 0}, upload=301419]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 01:56:37.151140589 (376h46m11.921781964s) [upload=301419]: Done uploading segments...
Jul 9 01:56:37.151169034 (376h46m11.921810369s) [upload=301419]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 20.079362496s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [..........................X......X..........X.............X.] duration:
40.811327674s
FAIL (2025-07-09 01:57:40 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
hRequestItem_SegmentBegin
Jul 9 01:56:00.629444016 (376h45m35.400085361s) [seg_pos={0 0}, upload=301417]: Begin upload segment...
Jul 9 01:56:00.629512196 (376h45m35.400153531s) [seg_pos={0 0}, upload=301417]: Done begin upload segment.
Jul 9 01:56:00.629572552 (376h45m35.400213887s) [noise=true , address=104.36.82.22:4532 , piece_id=3BRLUKXPLFBXIPKURV3BBSTA562UWWQBZ75HNYXEX75E74FKJIGQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.629579284 (376h45m35.400220619s) [noise=true , address=77.90.31.10:4539 , piece_id=OI7V6HU5WMYZSQCS5PAN6OUCAJY4UGUZVTESMLB4LHOQJVMKJLHQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.629690497 (376h45m35.400331832s) [noise=true , address=104.36.82.17:4333 , piece_id=TPALHRSYWC3XKW77CPA36EZOOJHPLUEN4V6DWFN5NHLIAFSN5KRQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630082818 (376h45m35.400724153s) [noise=true , address=104.36.82.6:4880 , piece_id=NKXAOUHYMQ4WR6V6ZGQOKRZWLAGI44AF4SNUR4OKUNJNICD4UUNQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630158242 (376h45m35.400799577s) [noise=true , address=79.127.201.214:4308 , piece_id=V4LWWG3RM4JTLZGNZ3XJDHEOZO7XW5R7BHQQJ6Z4Y47PZCEZHN6A, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630428680 (376h45m35.401070025s) [noise=true , address=77.90.31.12:4615 , piece_id=GLIW3756SYGXSX44VFMMDNYVEJPS6O4ZBNS53OJRV67QNGMGLPUA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630538129 (376h45m35.401179464s) [noise=true , address=104.36.82.60:4364 , piece_id=7WZJU4UIO4FLPUNWUH2PVIIH3ZWWVZCP2D2UUSCNBCIS727VVWPA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630862820 (376h45m35.401504165s) [noise=true , address=104.36.82.38:4708 , piece_id=RDIWOSET3Q2JKHRQ7HVUSVMNILHVEDQ7XYDL5DMWK37Q7YTRADQA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.631210084 (376h45m35.401851419s) [noise=true , address=104.36.82.21:4508 , piece_id=VVX5U234EZZNCZWIVYJVYBFDYG2SCKWS5HT5GM4H7CNNJQXDHT3Q, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.631555495 (376h45m35.40219684s) [noise=true , address=77.90.31.16:4743 , piece_id=2VSOXKLN6Q434ET7JYLRSU6BW7PIKNZ43MZOQ4HQL6MDP3WUJOZA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.631989165 (376h45m35.4026305s) [noise=true , address=104.36.82.23:4473 , piece_id=V3RAX7LZBU744EQQXDEG6Z25UUGQUFSSQCWCXNZKNGCTHD375GCA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.632258450 (376h45m35.402899785s) [noise=true , address=77.90.31.19:4850 , piece_id=DNYDFQHORBVHQTPKFLWEGQF6QRLE6IIE6YGTRP6SS63CPWVKRFFA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.632521984 (376h45m35.403163309s) [noise=true , address=45.91.116.5:4322 , piece_id=QMOE7Z74V4HHQO7LIN6KAJHEUSDKIAVC4XLFH3AOJMC2ILP6I4FA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.632835304 (376h45m35.403476649s) [noise=true , address=104.36.82.3:4791 , piece_id=NG7KRWBROBZWNP63A4734KWSRNGFVGLLXFZ5PGXZDJXFDTZNCZLQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.633120529 (376h45m35.403761864s) [noise=true , address=77.90.31.14:4692 , piece_id=NHO2352W7MMVGRS3MNPWSEFHWBOK73SQ6JQJEWZ5TV5UDFY44H5Q, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.636119667 (376h45m35.406761002s) [noise=true , address=104.36.82.2:4757 , piece_id=JMJIECO7WCFHWASHRCTEAG6VCU4S25SEETORWGJJQ63VAW533YJA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.636833744 (376h45m35.407475079s) [noise=true , address=77.90.31.15:4712 , piece_id=3DN3NPUYRN4AIBYBV2UCGE4TTYXD2WJRLHDMQEFIU36HBTYDOJ5Q, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.637112587 (376h45m35.407753922s) [noise=true , address=77.90.31.13:4643 , piece_id=PO24YVCOVVNORI7NRD3CDUC4ZPRULAXUESGBEDHJZYYDJ2ZUPQDA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:02.137197163 (376h45m36.907838498s) [upload=301417]: Next returned nil segment
Jul 9 01:56:02.137214166 (376h45m36.907855501s) [upload=301417]: Waiting for error group managing segments...
Jul 9 01:56:02.521317983 (376h45m37.291959318s) [noise=true , address=79.127.201.214:4308 , piece_id=V4LWWG3RM4JTLZGNZ3XJDHEOZO7XW5R7BHQQJ6Z4Y47PZCEZHN6A, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.567664820 (376h45m37.338306156s) [noise=true , address=77.90.31.14:4692 , piece_id=NHO2352W7MMVGRS3MNPWSEFHWBOK73SQ6JQJEWZ5TV5UDFY44H5Q, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.568760968 (376h45m37.339402303s) [noise=true , address=77.90.31.10:4539 , piece_id=OI7V6HU5WMYZSQCS5PAN6OUCAJY4UGUZVTESMLB4LHOQJVMKJLHQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.573315612 (376h45m37.343956947s) [noise=true , address=77.90.31.15:4712 , piece_id=3DN3NPUYRN4AIBYBV2UCGE4TTYXD2WJRLHDMQEFIU36HBTYDOJ5Q, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.577761728 (376h45m37.348403053s) [noise=true , address=77.90.31.19:4850 , piece_id=DNYDFQHORBVHQTPKFLWEGQF6QRLE6IIE6YGTRP6SS63CPWVKRFFA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.579676210 (376h45m37.350317545s) [noise=true , address=77.90.31.12:4615 , piece_id=GLIW3756SYGXSX44VFMMDNYVEJPS6O4ZBNS53OJRV67QNGMGLPUA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.595175322 (376h45m37.365816657s) [noise=true , address=77.90.31.16:4743 , piece_id=2VSOXKLN6Q434ET7JYLRSU6BW7PIKNZ43MZOQ4HQL6MDP3WUJOZA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.601273909 (376h45m37.371915244s) [noise=true , address=77.90.31.13:4643 , piece_id=PO24YVCOVVNORI7NRD3CDUC4ZPRULAXUESGBEDHJZYYDJ2ZUPQDA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:07.782538378 (376h45m42.553179714s) [noise=true , address=45.91.116.5:4322 , piece_id=QMOE7Z74V4HHQO7LIN6KAJHEUSDKIAVC4XLFH3AOJMC2ILP6I4FA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:22.662088184 (376h45m57.432729519s) [noise=true , address=104.36.82.17:4333 , piece_id=TPALHRSYWC3XKW77CPA36EZOOJHPLUEN4V6DWFN5NHLIAFSN5KRQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:24.053592013 (376h45m58.824233348s) [noise=true , address=104.36.82.2:4757 , piece_id=JMJIECO7WCFHWASHRCTEAG6VCU4S25SEETORWGJJQ63VAW533YJA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:44.900784468 (376h46m19.671425793s) [noise=true , address=104.36.82.23:4473 , piece_id=V3RAX7LZBU744EQQXDEG6Z25UUGQUFSSQCWCXNZKNGCTHD375GCA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:47.726372911 (376h46m22.497014236s) [noise=true , address=104.36.82.60:4364 , piece_id=7WZJU4UIO4FLPUNWUH2PVIIH3ZWWVZCP2D2UUSCNBCIS727VVWPA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:56.047624046 (376h46m30.818265381s) [noise=true , address=104.36.82.3:4791 , piece_id=NG7KRWBROBZWNP63A4734KWSRNGFVGLLXFZ5PGXZDJXFDTZNCZLQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:56.047644345 (376h46m30.81828568s) [seg_pos={0 0}, upload=301417]: Segment reached optimal threshold of 14 pieces.
Jul 9 01:56:56.047718136 (376h46m30.818359471s) [noise=true , address=104.36.82.38:4708 , piece_id=RDIWOSET3Q2JKHRQ7HVUSVMNILHVEDQ7XYDL5DMWK37Q7YTRADQA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BdJtXzvjMoNoQUjtHETyADPcZBTzcFzH8YDd651t7eTiFb1wf): protocol: expected piece hash; context canceled; EOF
Jul 9 01:56:56.047756840 (376h46m30.818398175s) [noise=true , address=104.36.82.22:4532 , piece_id=3BRLUKXPLFBXIPKURV3BBSTA562UWWQBZ75HNYXEX75E74FKJIGQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cHUuyc42zpw5k9Lz5Qo9AW9DMvKFPxRNdKZDZtjFaeyzT4Nkp): protocol: expected piece hash; context canceled; EOF
Jul 9 01:56:56.047948587 (376h46m30.818589922s) [noise=true , address=104.36.82.21:4508 , piece_id=VVX5U234EZZNCZWIVYJVYBFDYG2SCKWS5HT5GM4H7CNNJQXDHT3Q, seg_pos={0 0}, upload=301417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1269NRnqpabU6H8q9vSCmE8yMzznD1L7dyZFmzCWhtzJaTjbXg3): protocol: write tcp 188.166.225.170:60172->104.36.82.21:4508: use of closed network connection; write tcp 188.166.225.170:60172->104.36.82.21:4508: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60172->104.36.82.21:4508: use of closed network connection
Jul 9 01:56:56.047987552 (376h46m30.818628887s) [noise=true , address=104.36.82.6:4880 , piece_id=NKXAOUHYMQ4WR6V6ZGQOKRZWLAGI44AF4SNUR4OKUNJNICD4UUNQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125NsQhq7kRmTjKS8nuc51PZddApFEi5C38FzC5pEX2QrWYKzSW): protocol: write tcp 188.166.225.170:38944->104.36.82.6:4880: use of closed network connection; write tcp 188.166.225.170:38944->104.36.82.6:4880: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38944->104.36.82.6:4880: use of closed network connection
Jul 9 01:56:56.047996138 (376h46m30.818637463s) [seg_pos={0 0}, upload=301417]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 01:56:56.048046284 (376h46m30.818687619s) [upload=301417]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 01:56:56.048050091 (376h46m30.818691416s) [upload=301417]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 01:56:56.339173192 (376h46m31.109814527s) [upload=301417]: Done uploading segments...
Jul 9 01:56:56.339190416 (376h46m31.109831751s) [upload=301417]: Upload finished. err: <nil>
Duration: 40.8113512s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
15 out of 60 is failed (threshold=3) [X.X....X...X.X..X...X..X............X......X..X...X.X.X.X...] duration:
13.161340113s
FAIL (2025-07-09 01:57:21 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 13.161360361s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [X..........................X......X..........X.............X] duration:
1m0.002391837s
FAIL (2025-07-09 01:56:59 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
hRequestItem_SegmentBegin
Jul 9 01:56:00.629444016 (376h45m35.400085361s) [seg_pos={0 0}, upload=301417]: Begin upload segment...
Jul 9 01:56:00.629512196 (376h45m35.400153531s) [seg_pos={0 0}, upload=301417]: Done begin upload segment.
Jul 9 01:56:00.629572552 (376h45m35.400213887s) [noise=true , address=104.36.82.22:4532 , piece_id=3BRLUKXPLFBXIPKURV3BBSTA562UWWQBZ75HNYXEX75E74FKJIGQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.629579284 (376h45m35.400220619s) [noise=true , address=77.90.31.10:4539 , piece_id=OI7V6HU5WMYZSQCS5PAN6OUCAJY4UGUZVTESMLB4LHOQJVMKJLHQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.629690497 (376h45m35.400331832s) [noise=true , address=104.36.82.17:4333 , piece_id=TPALHRSYWC3XKW77CPA36EZOOJHPLUEN4V6DWFN5NHLIAFSN5KRQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630082818 (376h45m35.400724153s) [noise=true , address=104.36.82.6:4880 , piece_id=NKXAOUHYMQ4WR6V6ZGQOKRZWLAGI44AF4SNUR4OKUNJNICD4UUNQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630158242 (376h45m35.400799577s) [noise=true , address=79.127.201.214:4308 , piece_id=V4LWWG3RM4JTLZGNZ3XJDHEOZO7XW5R7BHQQJ6Z4Y47PZCEZHN6A, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630428680 (376h45m35.401070025s) [noise=true , address=77.90.31.12:4615 , piece_id=GLIW3756SYGXSX44VFMMDNYVEJPS6O4ZBNS53OJRV67QNGMGLPUA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630538129 (376h45m35.401179464s) [noise=true , address=104.36.82.60:4364 , piece_id=7WZJU4UIO4FLPUNWUH2PVIIH3ZWWVZCP2D2UUSCNBCIS727VVWPA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630862820 (376h45m35.401504165s) [noise=true , address=104.36.82.38:4708 , piece_id=RDIWOSET3Q2JKHRQ7HVUSVMNILHVEDQ7XYDL5DMWK37Q7YTRADQA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.631210084 (376h45m35.401851419s) [noise=true , address=104.36.82.21:4508 , piece_id=VVX5U234EZZNCZWIVYJVYBFDYG2SCKWS5HT5GM4H7CNNJQXDHT3Q, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.631555495 (376h45m35.40219684s) [noise=true , address=77.90.31.16:4743 , piece_id=2VSOXKLN6Q434ET7JYLRSU6BW7PIKNZ43MZOQ4HQL6MDP3WUJOZA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.631989165 (376h45m35.4026305s) [noise=true , address=104.36.82.23:4473 , piece_id=V3RAX7LZBU744EQQXDEG6Z25UUGQUFSSQCWCXNZKNGCTHD375GCA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.632258450 (376h45m35.402899785s) [noise=true , address=77.90.31.19:4850 , piece_id=DNYDFQHORBVHQTPKFLWEGQF6QRLE6IIE6YGTRP6SS63CPWVKRFFA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.632521984 (376h45m35.403163309s) [noise=true , address=45.91.116.5:4322 , piece_id=QMOE7Z74V4HHQO7LIN6KAJHEUSDKIAVC4XLFH3AOJMC2ILP6I4FA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.632835304 (376h45m35.403476649s) [noise=true , address=104.36.82.3:4791 , piece_id=NG7KRWBROBZWNP63A4734KWSRNGFVGLLXFZ5PGXZDJXFDTZNCZLQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.633120529 (376h45m35.403761864s) [noise=true , address=77.90.31.14:4692 , piece_id=NHO2352W7MMVGRS3MNPWSEFHWBOK73SQ6JQJEWZ5TV5UDFY44H5Q, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.636119667 (376h45m35.406761002s) [noise=true , address=104.36.82.2:4757 , piece_id=JMJIECO7WCFHWASHRCTEAG6VCU4S25SEETORWGJJQ63VAW533YJA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.636833744 (376h45m35.407475079s) [noise=true , address=77.90.31.15:4712 , piece_id=3DN3NPUYRN4AIBYBV2UCGE4TTYXD2WJRLHDMQEFIU36HBTYDOJ5Q, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.637112587 (376h45m35.407753922s) [noise=true , address=77.90.31.13:4643 , piece_id=PO24YVCOVVNORI7NRD3CDUC4ZPRULAXUESGBEDHJZYYDJ2ZUPQDA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:02.137197163 (376h45m36.907838498s) [upload=301417]: Next returned nil segment
Jul 9 01:56:02.137214166 (376h45m36.907855501s) [upload=301417]: Waiting for error group managing segments...
Jul 9 01:56:02.521317983 (376h45m37.291959318s) [noise=true , address=79.127.201.214:4308 , piece_id=V4LWWG3RM4JTLZGNZ3XJDHEOZO7XW5R7BHQQJ6Z4Y47PZCEZHN6A, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.567664820 (376h45m37.338306156s) [noise=true , address=77.90.31.14:4692 , piece_id=NHO2352W7MMVGRS3MNPWSEFHWBOK73SQ6JQJEWZ5TV5UDFY44H5Q, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.568760968 (376h45m37.339402303s) [noise=true , address=77.90.31.10:4539 , piece_id=OI7V6HU5WMYZSQCS5PAN6OUCAJY4UGUZVTESMLB4LHOQJVMKJLHQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.573315612 (376h45m37.343956947s) [noise=true , address=77.90.31.15:4712 , piece_id=3DN3NPUYRN4AIBYBV2UCGE4TTYXD2WJRLHDMQEFIU36HBTYDOJ5Q, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.577761728 (376h45m37.348403053s) [noise=true , address=77.90.31.19:4850 , piece_id=DNYDFQHORBVHQTPKFLWEGQF6QRLE6IIE6YGTRP6SS63CPWVKRFFA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.579676210 (376h45m37.350317545s) [noise=true , address=77.90.31.12:4615 , piece_id=GLIW3756SYGXSX44VFMMDNYVEJPS6O4ZBNS53OJRV67QNGMGLPUA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.595175322 (376h45m37.365816657s) [noise=true , address=77.90.31.16:4743 , piece_id=2VSOXKLN6Q434ET7JYLRSU6BW7PIKNZ43MZOQ4HQL6MDP3WUJOZA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.601273909 (376h45m37.371915244s) [noise=true , address=77.90.31.13:4643 , piece_id=PO24YVCOVVNORI7NRD3CDUC4ZPRULAXUESGBEDHJZYYDJ2ZUPQDA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:07.782538378 (376h45m42.553179714s) [noise=true , address=45.91.116.5:4322 , piece_id=QMOE7Z74V4HHQO7LIN6KAJHEUSDKIAVC4XLFH3AOJMC2ILP6I4FA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:22.662088184 (376h45m57.432729519s) [noise=true , address=104.36.82.17:4333 , piece_id=TPALHRSYWC3XKW77CPA36EZOOJHPLUEN4V6DWFN5NHLIAFSN5KRQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:24.053592013 (376h45m58.824233348s) [noise=true , address=104.36.82.2:4757 , piece_id=JMJIECO7WCFHWASHRCTEAG6VCU4S25SEETORWGJJQ63VAW533YJA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:44.900784468 (376h46m19.671425793s) [noise=true , address=104.36.82.23:4473 , piece_id=V3RAX7LZBU744EQQXDEG6Z25UUGQUFSSQCWCXNZKNGCTHD375GCA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:47.726372911 (376h46m22.497014236s) [noise=true , address=104.36.82.60:4364 , piece_id=7WZJU4UIO4FLPUNWUH2PVIIH3ZWWVZCP2D2UUSCNBCIS727VVWPA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:56.047624046 (376h46m30.818265381s) [noise=true , address=104.36.82.3:4791 , piece_id=NG7KRWBROBZWNP63A4734KWSRNGFVGLLXFZ5PGXZDJXFDTZNCZLQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:56.047644345 (376h46m30.81828568s) [seg_pos={0 0}, upload=301417]: Segment reached optimal threshold of 14 pieces.
Jul 9 01:56:56.047718136 (376h46m30.818359471s) [noise=true , address=104.36.82.38:4708 , piece_id=RDIWOSET3Q2JKHRQ7HVUSVMNILHVEDQ7XYDL5DMWK37Q7YTRADQA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BdJtXzvjMoNoQUjtHETyADPcZBTzcFzH8YDd651t7eTiFb1wf): protocol: expected piece hash; context canceled; EOF
Jul 9 01:56:56.047756840 (376h46m30.818398175s) [noise=true , address=104.36.82.22:4532 , piece_id=3BRLUKXPLFBXIPKURV3BBSTA562UWWQBZ75HNYXEX75E74FKJIGQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cHUuyc42zpw5k9Lz5Qo9AW9DMvKFPxRNdKZDZtjFaeyzT4Nkp): protocol: expected piece hash; context canceled; EOF
Jul 9 01:56:56.047948587 (376h46m30.818589922s) [noise=true , address=104.36.82.21:4508 , piece_id=VVX5U234EZZNCZWIVYJVYBFDYG2SCKWS5HT5GM4H7CNNJQXDHT3Q, seg_pos={0 0}, upload=301417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1269NRnqpabU6H8q9vSCmE8yMzznD1L7dyZFmzCWhtzJaTjbXg3): protocol: write tcp 188.166.225.170:60172->104.36.82.21:4508: use of closed network connection; write tcp 188.166.225.170:60172->104.36.82.21:4508: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60172->104.36.82.21:4508: use of closed network connection
Jul 9 01:56:56.047987552 (376h46m30.818628887s) [noise=true , address=104.36.82.6:4880 , piece_id=NKXAOUHYMQ4WR6V6ZGQOKRZWLAGI44AF4SNUR4OKUNJNICD4UUNQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125NsQhq7kRmTjKS8nuc51PZddApFEi5C38FzC5pEX2QrWYKzSW): protocol: write tcp 188.166.225.170:38944->104.36.82.6:4880: use of closed network connection; write tcp 188.166.225.170:38944->104.36.82.6:4880: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38944->104.36.82.6:4880: use of closed network connection
Jul 9 01:56:56.047996138 (376h46m30.818637463s) [seg_pos={0 0}, upload=301417]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 01:56:56.048046284 (376h46m30.818687619s) [upload=301417]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 01:56:56.048050091 (376h46m30.818691416s) [upload=301417]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 01:56:56.339173192 (376h46m31.109814527s) [upload=301417]: Done uploading segments...
Jul 9 01:56:56.339190416 (376h46m31.109831751s) [upload=301417]: Upload finished. err: <nil>
Duration: 1m0.002406606s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
16 out of 60 is failed (threshold=3) [..............X......XX.XX..X......XXX....XX.....XX...X.XX..] duration:
35.861354975s
FAIL (2025-07-09 01:56:56 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 35.861377038s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.................X......................X.................X.] duration:
5.689003624s
FAIL (2025-07-09 01:56:43 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
em: *pb.BatchRequestItem_SegmentCommit
Jul 9 01:55:26.378238935 (376h45m1.14888026s) [upload=301404]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 01:55:26.681901575 (376h45m1.45254291s) [upload=301404]: Done uploading segments...
Jul 9 01:55:26.681918448 (376h45m1.452559783s) [upload=301404]: Upload finished. err: <nil>
Jul 9 01:56:07.557068843 (376h45m42.327710178s) [upload=301419]: Starting upload
Jul 9 01:56:07.557123908 (376h45m42.327765243s) [upload=301419]: Done starting upload
Jul 9 01:56:07.557294845 (376h45m42.32793618s) [upload=301419]: Uploading segments...
Jul 9 01:56:07.557334942 (376h45m42.327976267s) [upload=301419]: Got next segment. Inline: false
Jul 9 01:56:07.557358848 (376h45m42.328000183s) [upload=301419]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 01:56:07.557363737 (376h45m42.328005072s) [upload=301419]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 01:56:07.786768822 (376h45m42.557410157s) [seg_pos={0 0}, upload=301419]: Begin upload segment...
Jul 9 01:56:07.786807697 (376h45m42.557449032s) [seg_pos={0 0}, upload=301419]: Done begin upload segment.
Jul 9 01:56:07.786827744 (376h45m42.557469069s) [noise=true , address=45.91.116.13:4154 , piece_id=WT7WL2P7RMJFA7XIZ3KNRVJ2RWDQGQBCM3J65X2X3MQ6WSAKHOIQ, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.786867240 (376h45m42.557508576s) [noise=true , address=104.36.82.2:4732 , piece_id=HF35AWQTHK4B5XQXLUOM5HPEBINSUIXGA3AAMZFEHZW4GDRDT4KQ, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.787161643 (376h45m42.557802968s) [noise=true , address=104.36.82.16:4294 , piece_id=TWDA4QZ5WAQF3OOXDDNAHBKYM5OAKBZ34QWW2FYXHJ6KGKDTDFRQ, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.787243289 (376h45m42.557884614s) [noise=true , address=77.90.31.11:4555 , piece_id=A7UGOPRF44445XR4OBEODVMMJU4HED7JUOPORHJKXVEPJO6KYIWQ, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.787405490 (376h45m42.558046815s) [noise=true , address=77.90.31.10:4518 , piece_id=ZC7HW4EDPE6UU2MLOHGHMYKBA4NXTJEL4I6CP6FK62QIAOEUT4ZQ, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.787501334 (376h45m42.558142669s) [noise=true , address=45.91.116.6:4975 , piece_id=MRETRUDTUOXXPHG4P25RMAJP4EDX4SS4B7O22PPPIQBSULZ52KNA, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.787663624 (376h45m42.558304959s) [noise=true , address=104.36.82.4:4801 , piece_id=OVVFQSK4TYRAXFNR6NZ74463CDC5BPKVROPU45ATGOTCZDUFDWHQ, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.787980670 (376h45m42.558622005s) [noise=true , address=45.91.116.11:4117 , piece_id=VDNXPWCQBXOOEAEJB7YB75AWP25A2AMCLTOT6KHFETFUXZ6UH3IQ, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.788245387 (376h45m42.558886712s) [noise=true , address=104.36.82.10:4078 , piece_id=ORPJSLOMY4IAB2NWYOQ7RCZCLEIQ25CIXFKKTWW5LBO5BFPPY64Q, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.788473754 (376h45m42.559115079s) [noise=true , address=45.91.116.7:4009 , piece_id=EEGGWRYW7EDJOVZDWO36KAGPFBQFSK5FJ7YU2FPBBWSIGV2KM4UA, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.788656052 (376h45m42.559297387s) [noise=true , address=104.36.82.6:4876 , piece_id=TUZF2S3DJSUK6MMNEQYWR35I56NAFLDRFPL646TIAGIUNVWE2ZGA, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.788848981 (376h45m42.559490316s) [noise=true , address=45.91.116.10:4081 , piece_id=4ZKRZFKWSILUJ3QNHIDRBGUETYMO7UZI5G5XR64XI2ANHXRCKAZQ, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.789032262 (376h45m42.559673597s) [noise=true , address=104.36.82.9:4006 , piece_id=P7YEMWPREVLDDFE7JDYAOEZJIYCAL3OAKKJJDC6EBMLLYSVRL73Q, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.789272181 (376h45m42.559913506s) [noise=true , address=77.90.31.19:4843 , piece_id=5FRTA2EJNZVNHV47EWYRFUH6VW2FJN2EBAKBRJWMP66BXF6SZF6A, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.789459429 (376h45m42.560100764s) [noise=true , address=45.91.116.8:4047 , piece_id=VISW3TARCTQMHB3662HBA2I7UW4IIREOJ32XOSBNTRPPTB4KD77A, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.789640225 (376h45m42.56028155s) [noise=true , address=77.90.31.21:4877 , piece_id=EN7QS3CEXXIBY7FU6Y32LLDKYCSVRJQ4G7RXRE2DV7CUPMZBVHDQ, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:09.422697440 (376h45m44.193338775s) [upload=301419]: Next returned nil segment
Jul 9 01:56:09.422714812 (376h45m44.193356148s) [upload=301419]: Waiting for error group managing segments...
Jul 9 01:56:09.703007945 (376h45m44.47364928s) [noise=true , address=104.36.82.9:4006 , piece_id=P7YEMWPREVLDDFE7JDYAOEZJIYCAL3OAKKJJDC6EBMLLYSVRL73Q, seg_pos={0 0}, upload=301419]: Done uploading piece. err: <nil>
Jul 9 01:56:09.780867601 (376h45m44.551508936s) [noise=true , address=104.36.82.16:4294 , piece_id=TWDA4QZ5WAQF3OOXDDNAHBKYM5OAKBZ34QWW2FYXHJ6KGKDTDFRQ, seg_pos={0 0}, upload=301419]: Done uploading piece. err: <nil>
Jul 9 01:56:09.840231843 (376h45m44.610873178s) [noise=true , address=45.91.116.6:4975 , piece_id=MRETRUDTUOXXPHG4P25RMAJP4EDX4SS4B7O22PPPIQBSULZ52KNA, seg_pos={0 0}, upload=301419]: Done uploading piece. err: <nil>
Jul 9 01:56:09.841984998 (376h45m44.612626363s) [noise=true , address=45.91.116.10:4081 , piece_id=4ZKRZFKWSILUJ3QNHIDRBGUETYMO7UZI5G5XR64XI2ANHXRCKAZQ, seg_pos={0 0}, upload=301419]: Done uploading piece. err: <nil>
Jul 9 01:56:09.850995516 (376h45m44.621636851s) [noise=true , address=45.91.116.13:4154 , piece_id=WT7WL2P7RMJFA7XIZ3KNRVJ2RWDQGQBCM3J65X2X3MQ6WSAKHOIQ, seg_pos={0 0}, upload=301419]: Done uploading piece. err: <nil>
Jul 9 01:56:09.860567239 (376h45m44.631208574s) [noise=true , address=45.91.116.7:4009 , piece_id=EEGGWRYW7EDJOVZDWO36KAGPFBQFSK5FJ7YU2FPBBWSIGV2KM4UA, seg_pos={0 0}, upload=301419]: Done uploading piece. err: <nil>
Jul 9 01:56:09.871129736 (376h45m44.641771071s) [noise=true , address=77.90.31.19:4843 , piece_id=5FRTA2EJNZVNHV47EWYRFUH6VW2FJN2EBAKBRJWMP66BXF6SZF6A, seg_pos={0 0}, upload=301419]: Done uploading piece. err: <nil>
Jul 9 01:56:09.874489343 (376h45m44.645130678s) [noise=true , address=77.90.31.11:4555 , piece_id=A7UGOPRF44445XR4OBEODVMMJU4HED7JUOPORHJKXVEPJO6KYIWQ, seg_pos={0 0}, upload=301419]: Done uploading piece. err: <nil>
Jul 9 01:56:09.875282992 (376h45m44.645924327s) [noise=true , address=77.90.31.21:4877 , piece_id=EN7QS3CEXXIBY7FU6Y32LLDKYCSVRJQ4G7RXRE2DV7CUPMZBVHDQ, seg_pos={0 0}, upload=301419]: Done uploading piece. err: <nil>
Jul 9 01:56:09.882277543 (376h45m44.652918878s) [noise=true , address=77.90.31.10:4518 , piece_id=ZC7HW4EDPE6UU2MLOHGHMYKBA4NXTJEL4I6CP6FK62QIAOEUT4ZQ, seg_pos={0 0}, upload=301419]: Done uploading piece. err: <nil>
Jul 9 01:56:09.909865152 (376h45m44.680506487s) [noise=true , address=45.91.116.11:4117 , piece_id=VDNXPWCQBXOOEAEJB7YB75AWP25A2AMCLTOT6KHFETFUXZ6UH3IQ, seg_pos={0 0}, upload=301419]: Done uploading piece. err: <nil>
Jul 9 01:56:09.930049268 (376h45m44.700690603s) [noise=true , address=45.91.116.8:4047 , piece_id=VISW3TARCTQMHB3662HBA2I7UW4IIREOJ32XOSBNTRPPTB4KD77A, seg_pos={0 0}, upload=301419]: Done uploading piece. err: <nil>
Jul 9 01:56:36.916221275 (376h46m11.68686263s) [noise=true , address=104.36.82.6:4876 , piece_id=TUZF2S3DJSUK6MMNEQYWR35I56NAFLDRFPL646TIAGIUNVWE2ZGA, seg_pos={0 0}, upload=301419]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 188.166.225.170:36068->104.36.82.6:4876: use of closed network connection; write tcp 188.166.225.170:36068->104.36.82.6:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36068->104.36.82.6:4876: use of closed network connection
Jul 9 01:56:36.916318731 (376h46m11.686960076s) [noise=true , address=104.36.82.4:4801 , piece_id=OVVFQSK4TYRAXFNR6NZ74463CDC5BPKVROPU45ATGOTCZDUFDWHQ, seg_pos={0 0}, upload=301419]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 188.166.225.170:36588->104.36.82.4:4801: use of closed network connection; context deadline exceeded
Jul 9 01:56:36.916361463 (376h46m11.687002818s) [noise=true , address=104.36.82.2:4732 , piece_id=HF35AWQTHK4B5XQXLUOM5HPEBINSUIXGA3AAMZFEHZW4GDRDT4KQ, seg_pos={0 0}, upload=301419]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: write tcp 188.166.225.170:44126->104.36.82.2:4732: use of closed network connection; write tcp 188.166.225.170:44126->104.36.82.2:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44126->104.36.82.2:4732: use of closed network connection
Jul 9 01:56:36.916428551 (376h46m11.687069916s) [noise=true , address=104.36.82.10:4078 , piece_id=ORPJSLOMY4IAB2NWYOQ7RCZCLEIQ25CIXFKKTWW5LBO5BFPPY64Q, seg_pos={0 0}, upload=301419]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 188.166.225.170:53862->104.36.82.10:4078: use of closed network connection; write tcp 188.166.225.170:53862->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53862->104.36.82.10:4078: use of closed network connection
Jul 9 01:56:36.916470932 (376h46m11.687112338s) [seg_pos={0 0}, upload=301419]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 01:56:37.151140589 (376h46m11.921781964s) [upload=301419]: Done uploading segments...
Jul 9 01:56:37.151169034 (376h46m11.921810369s) [upload=301419]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.689024253s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..................X......................X.................X] duration:
30.23686255s
FAIL (2025-07-09 01:56:37 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
em: *pb.BatchRequestItem_SegmentCommit
Jul 9 01:55:26.378238935 (376h45m1.14888026s) [upload=301404]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 01:55:26.681901575 (376h45m1.45254291s) [upload=301404]: Done uploading segments...
Jul 9 01:55:26.681918448 (376h45m1.452559783s) [upload=301404]: Upload finished. err: <nil>
Jul 9 01:56:07.557068843 (376h45m42.327710178s) [upload=301419]: Starting upload
Jul 9 01:56:07.557123908 (376h45m42.327765243s) [upload=301419]: Done starting upload
Jul 9 01:56:07.557294845 (376h45m42.32793618s) [upload=301419]: Uploading segments...
Jul 9 01:56:07.557334942 (376h45m42.327976267s) [upload=301419]: Got next segment. Inline: false
Jul 9 01:56:07.557358848 (376h45m42.328000183s) [upload=301419]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 01:56:07.557363737 (376h45m42.328005072s) [upload=301419]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 01:56:07.786768822 (376h45m42.557410157s) [seg_pos={0 0}, upload=301419]: Begin upload segment...
Jul 9 01:56:07.786807697 (376h45m42.557449032s) [seg_pos={0 0}, upload=301419]: Done begin upload segment.
Jul 9 01:56:07.786827744 (376h45m42.557469069s) [noise=true , address=45.91.116.13:4154 , piece_id=WT7WL2P7RMJFA7XIZ3KNRVJ2RWDQGQBCM3J65X2X3MQ6WSAKHOIQ, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.786867240 (376h45m42.557508576s) [noise=true , address=104.36.82.2:4732 , piece_id=HF35AWQTHK4B5XQXLUOM5HPEBINSUIXGA3AAMZFEHZW4GDRDT4KQ, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.787161643 (376h45m42.557802968s) [noise=true , address=104.36.82.16:4294 , piece_id=TWDA4QZ5WAQF3OOXDDNAHBKYM5OAKBZ34QWW2FYXHJ6KGKDTDFRQ, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.787243289 (376h45m42.557884614s) [noise=true , address=77.90.31.11:4555 , piece_id=A7UGOPRF44445XR4OBEODVMMJU4HED7JUOPORHJKXVEPJO6KYIWQ, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.787405490 (376h45m42.558046815s) [noise=true , address=77.90.31.10:4518 , piece_id=ZC7HW4EDPE6UU2MLOHGHMYKBA4NXTJEL4I6CP6FK62QIAOEUT4ZQ, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.787501334 (376h45m42.558142669s) [noise=true , address=45.91.116.6:4975 , piece_id=MRETRUDTUOXXPHG4P25RMAJP4EDX4SS4B7O22PPPIQBSULZ52KNA, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.787663624 (376h45m42.558304959s) [noise=true , address=104.36.82.4:4801 , piece_id=OVVFQSK4TYRAXFNR6NZ74463CDC5BPKVROPU45ATGOTCZDUFDWHQ, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.787980670 (376h45m42.558622005s) [noise=true , address=45.91.116.11:4117 , piece_id=VDNXPWCQBXOOEAEJB7YB75AWP25A2AMCLTOT6KHFETFUXZ6UH3IQ, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.788245387 (376h45m42.558886712s) [noise=true , address=104.36.82.10:4078 , piece_id=ORPJSLOMY4IAB2NWYOQ7RCZCLEIQ25CIXFKKTWW5LBO5BFPPY64Q, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.788473754 (376h45m42.559115079s) [noise=true , address=45.91.116.7:4009 , piece_id=EEGGWRYW7EDJOVZDWO36KAGPFBQFSK5FJ7YU2FPBBWSIGV2KM4UA, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.788656052 (376h45m42.559297387s) [noise=true , address=104.36.82.6:4876 , piece_id=TUZF2S3DJSUK6MMNEQYWR35I56NAFLDRFPL646TIAGIUNVWE2ZGA, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.788848981 (376h45m42.559490316s) [noise=true , address=45.91.116.10:4081 , piece_id=4ZKRZFKWSILUJ3QNHIDRBGUETYMO7UZI5G5XR64XI2ANHXRCKAZQ, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.789032262 (376h45m42.559673597s) [noise=true , address=104.36.82.9:4006 , piece_id=P7YEMWPREVLDDFE7JDYAOEZJIYCAL3OAKKJJDC6EBMLLYSVRL73Q, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.789272181 (376h45m42.559913506s) [noise=true , address=77.90.31.19:4843 , piece_id=5FRTA2EJNZVNHV47EWYRFUH6VW2FJN2EBAKBRJWMP66BXF6SZF6A, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.789459429 (376h45m42.560100764s) [noise=true , address=45.91.116.8:4047 , piece_id=VISW3TARCTQMHB3662HBA2I7UW4IIREOJ32XOSBNTRPPTB4KD77A, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.789640225 (376h45m42.56028155s) [noise=true , address=77.90.31.21:4877 , piece_id=EN7QS3CEXXIBY7FU6Y32LLDKYCSVRJQ4G7RXRE2DV7CUPMZBVHDQ, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:09.422697440 (376h45m44.193338775s) [upload=301419]: Next returned nil segment
Jul 9 01:56:09.422714812 (376h45m44.193356148s) [upload=301419]: Waiting for error group managing segments...
Jul 9 01:56:09.703007945 (376h45m44.47364928s) [noise=true , address=104.36.82.9:4006 , piece_id=P7YEMWPREVLDDFE7JDYAOEZJIYCAL3OAKKJJDC6EBMLLYSVRL73Q, seg_pos={0 0}, upload=301419]: Done uploading piece. err: <nil>
Jul 9 01:56:09.780867601 (376h45m44.551508936s) [noise=true , address=104.36.82.16:4294 , piece_id=TWDA4QZ5WAQF3OOXDDNAHBKYM5OAKBZ34QWW2FYXHJ6KGKDTDFRQ, seg_pos={0 0}, upload=301419]: Done uploading piece. err: <nil>
Jul 9 01:56:09.840231843 (376h45m44.610873178s) [noise=true , address=45.91.116.6:4975 , piece_id=MRETRUDTUOXXPHG4P25RMAJP4EDX4SS4B7O22PPPIQBSULZ52KNA, seg_pos={0 0}, upload=301419]: Done uploading piece. err: <nil>
Jul 9 01:56:09.841984998 (376h45m44.612626363s) [noise=true , address=45.91.116.10:4081 , piece_id=4ZKRZFKWSILUJ3QNHIDRBGUETYMO7UZI5G5XR64XI2ANHXRCKAZQ, seg_pos={0 0}, upload=301419]: Done uploading piece. err: <nil>
Jul 9 01:56:09.850995516 (376h45m44.621636851s) [noise=true , address=45.91.116.13:4154 , piece_id=WT7WL2P7RMJFA7XIZ3KNRVJ2RWDQGQBCM3J65X2X3MQ6WSAKHOIQ, seg_pos={0 0}, upload=301419]: Done uploading piece. err: <nil>
Jul 9 01:56:09.860567239 (376h45m44.631208574s) [noise=true , address=45.91.116.7:4009 , piece_id=EEGGWRYW7EDJOVZDWO36KAGPFBQFSK5FJ7YU2FPBBWSIGV2KM4UA, seg_pos={0 0}, upload=301419]: Done uploading piece. err: <nil>
Jul 9 01:56:09.871129736 (376h45m44.641771071s) [noise=true , address=77.90.31.19:4843 , piece_id=5FRTA2EJNZVNHV47EWYRFUH6VW2FJN2EBAKBRJWMP66BXF6SZF6A, seg_pos={0 0}, upload=301419]: Done uploading piece. err: <nil>
Jul 9 01:56:09.874489343 (376h45m44.645130678s) [noise=true , address=77.90.31.11:4555 , piece_id=A7UGOPRF44445XR4OBEODVMMJU4HED7JUOPORHJKXVEPJO6KYIWQ, seg_pos={0 0}, upload=301419]: Done uploading piece. err: <nil>
Jul 9 01:56:09.875282992 (376h45m44.645924327s) [noise=true , address=77.90.31.21:4877 , piece_id=EN7QS3CEXXIBY7FU6Y32LLDKYCSVRJQ4G7RXRE2DV7CUPMZBVHDQ, seg_pos={0 0}, upload=301419]: Done uploading piece. err: <nil>
Jul 9 01:56:09.882277543 (376h45m44.652918878s) [noise=true , address=77.90.31.10:4518 , piece_id=ZC7HW4EDPE6UU2MLOHGHMYKBA4NXTJEL4I6CP6FK62QIAOEUT4ZQ, seg_pos={0 0}, upload=301419]: Done uploading piece. err: <nil>
Jul 9 01:56:09.909865152 (376h45m44.680506487s) [noise=true , address=45.91.116.11:4117 , piece_id=VDNXPWCQBXOOEAEJB7YB75AWP25A2AMCLTOT6KHFETFUXZ6UH3IQ, seg_pos={0 0}, upload=301419]: Done uploading piece. err: <nil>
Jul 9 01:56:09.930049268 (376h45m44.700690603s) [noise=true , address=45.91.116.8:4047 , piece_id=VISW3TARCTQMHB3662HBA2I7UW4IIREOJ32XOSBNTRPPTB4KD77A, seg_pos={0 0}, upload=301419]: Done uploading piece. err: <nil>
Jul 9 01:56:36.916221275 (376h46m11.68686263s) [noise=true , address=104.36.82.6:4876 , piece_id=TUZF2S3DJSUK6MMNEQYWR35I56NAFLDRFPL646TIAGIUNVWE2ZGA, seg_pos={0 0}, upload=301419]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 188.166.225.170:36068->104.36.82.6:4876: use of closed network connection; write tcp 188.166.225.170:36068->104.36.82.6:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36068->104.36.82.6:4876: use of closed network connection
Jul 9 01:56:36.916318731 (376h46m11.686960076s) [noise=true , address=104.36.82.4:4801 , piece_id=OVVFQSK4TYRAXFNR6NZ74463CDC5BPKVROPU45ATGOTCZDUFDWHQ, seg_pos={0 0}, upload=301419]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 188.166.225.170:36588->104.36.82.4:4801: use of closed network connection; context deadline exceeded
Jul 9 01:56:36.916361463 (376h46m11.687002818s) [noise=true , address=104.36.82.2:4732 , piece_id=HF35AWQTHK4B5XQXLUOM5HPEBINSUIXGA3AAMZFEHZW4GDRDT4KQ, seg_pos={0 0}, upload=301419]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: write tcp 188.166.225.170:44126->104.36.82.2:4732: use of closed network connection; write tcp 188.166.225.170:44126->104.36.82.2:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44126->104.36.82.2:4732: use of closed network connection
Jul 9 01:56:36.916428551 (376h46m11.687069916s) [noise=true , address=104.36.82.10:4078 , piece_id=ORPJSLOMY4IAB2NWYOQ7RCZCLEIQ25CIXFKKTWW5LBO5BFPPY64Q, seg_pos={0 0}, upload=301419]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 188.166.225.170:53862->104.36.82.10:4078: use of closed network connection; write tcp 188.166.225.170:53862->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53862->104.36.82.10:4078: use of closed network connection
Jul 9 01:56:36.916470932 (376h46m11.687112338s) [seg_pos={0 0}, upload=301419]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 01:56:37.151140589 (376h46m11.921781964s) [upload=301419]: Done uploading segments...
Jul 9 01:56:37.151169034 (376h46m11.921810369s) [upload=301419]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 30.236882728s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
16 out of 60 is failed (threshold=3) [...............X......XX.XX..X......XXX....XX.....XX...X.XX.] duration:
57.178987413s
FAIL (2025-07-09 01:56:19 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 57.178998534s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
15 out of 60 is failed (threshold=3) [.X.X....X...X.X..X...X..X............X......X..X...X.X.X.X..] duration:
6.615274174s
FAIL (2025-07-09 01:56:02 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.615284644s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [.X..........................X......X..........X.............] duration:
41.407428263s
FAIL (2025-07-09 01:55:59 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
s) [upload=301248]: Got next segment. Inline: false
Jul 9 01:43:24.322645491 (376h32m59.093286826s) [upload=301248]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 01:43:24.322650441 (376h32m59.093291777s) [upload=301248]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 01:43:24.769774209 (376h32m59.540415544s) [seg_pos={0 0}, upload=301248]: Begin upload segment...
Jul 9 01:43:24.769830437 (376h32m59.540471772s) [seg_pos={0 0}, upload=301248]: Done begin upload segment.
Jul 9 01:43:24.769852048 (376h32m59.540493373s) [noise=true , address=79.127.205.234:4614 , piece_id=JPPMN7KOKXFYF2YDK4RC7AU6KYV2UHQELFEG6ZZ6W5HQ2MKFVN5A, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.769864432 (376h32m59.540505767s) [noise=true , address=45.91.116.11:4123 , piece_id=WSMWN6I2QVBTO6OCRRDCMT3G3TESW3H2ZET5YNWFUPVENPN3UWKQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.770250630 (376h32m59.540891965s) [noise=true , address=104.36.82.22:4511 , piece_id=3AIKVWFRA4CLMPRDYFNX6EA7XO22MB7RQEOVNXPUVUCNFAHPSFCQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.770361543 (376h32m59.541002878s) [noise=true , address=77.90.31.21:4886 , piece_id=YRDZYEV6KU5YISF4DIS3GTZVBFRCZEOIW2DG7VCKHO26NRXMHMVQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.770542389 (376h32m59.541183724s) [noise=true , address=77.90.31.19:4848 , piece_id=RV7TBRZW2YH64X5CLXFHFTJD2WQCEM3P7OW3NYZ2IWC2E3DJDG5A, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.770626239 (376h32m59.541267594s) [noise=true , address=104.36.82.19:4411 , piece_id=R27NIJUXHNTQWK4IXXSVS2QC535JAZJ3K2WV4VANVOGKRYV7YADA, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.770818237 (376h32m59.541459572s) [noise=true , address=104.36.82.40:4955 , piece_id=3TVDS5QR5267W6VKWNXOPW6ZSTXISGXFBIPKK7E3S6ML7C4X4MUQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.770918087 (376h32m59.541559422s) [noise=true , address=104.36.82.12:4160 , piece_id=2Z3FQL5GTT26HC2GPKO5YPOLPB2Z75K4UG3FDPMOEDCCD6Q6MULQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.771127618 (376h32m59.541768953s) [noise=true , address=77.90.31.14:4677 , piece_id=Z7SGXIO722E3U7PWA4AWOUD5T27VSQUCG3ZBVP22WJRFSRKOEXAQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.771146836 (376h32m59.541788171s) [noise=true , address=77.90.31.12:4622 , piece_id=25K7AOPYFLZ6EDAA3SPHHE6S635PN5VB3ZK4LIULBHKQGYYQ3C7A, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.771367858 (376h32m59.542009193s) [noise=true , address=77.90.31.18:4825 , piece_id=NFOYMUTMKMKTPCAEPBTWL43ISOETOAM4WKVAUNHCETNW75IQLLMQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.771421060 (376h32m59.542062395s) [noise=true , address=77.90.31.14:4691 , piece_id=ELMXSLQT6TNOMGNUGM4BEHZ2P7OZWNSX6EOCHDAYOD7OIANFUVHA, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.771586527 (376h32m59.542227852s) [noise=true , address=64.49.13.169:4706 , piece_id=2CKVJCKZJH4F3VC3X7XJOWMYHVHU52POGAA7VMAU4YM6HTVHC5PA, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.771699443 (376h32m59.542340778s) [noise=true , address=104.36.82.44:4898 , piece_id=X47T6NYNODFSVKHJGWIH2WJT7BA3UVP7BEBYVKPPQBMN4QOQCJ6Q, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.771818129 (376h32m59.542459465s) [noise=true , address=104.36.82.34:4551 , piece_id=HWYKZWYEKMJT3JHDPHRWIAFKP5NZQ72OGR4SPR2I2SRRABLA663A, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.772055795 (376h32m59.54269713s) [noise=true , address=77.90.31.10:4543 , piece_id=QCKPGFCSQ3N57BWXMOSW5LAVA4KOI3EVUFU2OMPN7CMAOYAH4RWQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.772089298 (376h32m59.542730623s) [noise=true , address=104.36.82.23:4459 , piece_id=SOG4BGCMVK36OVMKHGIYGSVGX2ZZ2PWZV74S2LQS6IZQRGDYJVYA, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.772274673 (376h32m59.542916008s) [noise=true , address=77.90.31.15:4715 , piece_id=ITPPGCX32ZLU72XXE7RG6AYE3PM7IILVNPXM5A4NJ2S7P6U4Y2FA, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:25.540057386 (376h33m0.310698721s) [noise=true , address=64.49.13.169:4706 , piece_id=2CKVJCKZJH4F3VC3X7XJOWMYHVHU52POGAA7VMAU4YM6HTVHC5PA, seg_pos={0 0}, upload=301248]: Done uploading piece. err: ecclient: upload failed (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7, address:64.49.13.169:4706): protocol: storage node overloaded, request limit: 2000
Jul 9 01:43:25.878055136 (376h33m0.648696481s) [upload=301248]: Next returned nil segment
Jul 9 01:43:25.878073130 (376h33m0.648714465s) [upload=301248]: Waiting for error group managing segments...
Jul 9 01:43:26.175673001 (376h33m0.946314336s) [noise=true , address=79.127.205.234:4614 , piece_id=JPPMN7KOKXFYF2YDK4RC7AU6KYV2UHQELFEG6ZZ6W5HQ2MKFVN5A, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.774276626 (376h33m1.544917961s) [noise=true , address=77.90.31.12:4622 , piece_id=25K7AOPYFLZ6EDAA3SPHHE6S635PN5VB3ZK4LIULBHKQGYYQ3C7A, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.778019557 (376h33m1.548660892s) [noise=true , address=77.90.31.21:4886 , piece_id=YRDZYEV6KU5YISF4DIS3GTZVBFRCZEOIW2DG7VCKHO26NRXMHMVQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.779099603 (376h33m1.549740938s) [noise=true , address=77.90.31.14:4691 , piece_id=ELMXSLQT6TNOMGNUGM4BEHZ2P7OZWNSX6EOCHDAYOD7OIANFUVHA, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.779161091 (376h33m1.549802426s) [noise=true , address=77.90.31.18:4825 , piece_id=NFOYMUTMKMKTPCAEPBTWL43ISOETOAM4WKVAUNHCETNW75IQLLMQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.780045593 (376h33m1.550686928s) [noise=true , address=77.90.31.14:4677 , piece_id=Z7SGXIO722E3U7PWA4AWOUD5T27VSQUCG3ZBVP22WJRFSRKOEXAQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.795740620 (376h33m1.566381976s) [noise=true , address=77.90.31.10:4543 , piece_id=QCKPGFCSQ3N57BWXMOSW5LAVA4KOI3EVUFU2OMPN7CMAOYAH4RWQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.795758073 (376h33m1.566399408s) [noise=true , address=77.90.31.15:4715 , piece_id=ITPPGCX32ZLU72XXE7RG6AYE3PM7IILVNPXM5A4NJ2S7P6U4Y2FA, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.812585147 (376h33m1.583226483s) [noise=true , address=77.90.31.19:4848 , piece_id=RV7TBRZW2YH64X5CLXFHFTJD2WQCEM3P7OW3NYZ2IWC2E3DJDG5A, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:27.386630478 (376h33m2.157271803s) [noise=true , address=45.91.116.11:4123 , piece_id=WSMWN6I2QVBTO6OCRRDCMT3G3TESW3H2ZET5YNWFUPVENPN3UWKQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:37.603914421 (376h33m12.374555756s) [noise=true , address=104.36.82.34:4551 , piece_id=HWYKZWYEKMJT3JHDPHRWIAFKP5NZQ72OGR4SPR2I2SRRABLA663A, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:41.629246242 (376h33m16.399887567s) [noise=true , address=104.36.82.12:4160 , piece_id=2Z3FQL5GTT26HC2GPKO5YPOLPB2Z75K4UG3FDPMOEDCCD6Q6MULQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:44:10.222815170 (376h33m44.993456505s) [noise=true , address=104.36.82.22:4511 , piece_id=3AIKVWFRA4CLMPRDYFNX6EA7XO22MB7RQEOVNXPUVUCNFAHPSFCQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:44:21.573027994 (376h33m56.343669329s) [noise=true , address=104.36.82.23:4459 , piece_id=SOG4BGCMVK36OVMKHGIYGSVGX2ZZ2PWZV74S2LQS6IZQRGDYJVYA, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:44:21.573054295 (376h33m56.34369563s) [seg_pos={0 0}, upload=301248]: Segment reached optimal threshold of 14 pieces.
Jul 9 01:44:21.573140269 (376h33m56.343781604s) [noise=true , address=104.36.82.40:4955 , piece_id=3TVDS5QR5267W6VKWNXOPW6ZSTXISGXFBIPKK7E3S6ML7C4X4MUQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ssLq5bo7og4z8ZPd28f2H6WUExsWohBBZUnciGaH5KKpcQ25d): protocol: expected piece hash; context canceled; EOF
Jul 9 01:44:21.573153795 (376h33m56.34379513s) [noise=true , address=104.36.82.19:4411 , piece_id=R27NIJUXHNTQWK4IXXSVS2QC535JAZJ3K2WV4VANVOGKRYV7YADA, seg_pos={0 0}, upload=301248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nradx4SMYLkuJw3Ad6o9Sq1iNtQR6GDu8qg7tCzZzNcTjVfyd): protocol: expected piece hash; context canceled; EOF
Jul 9 01:44:21.573259487 (376h33m56.343900822s) [noise=true , address=104.36.82.44:4898 , piece_id=X47T6NYNODFSVKHJGWIH2WJT7BA3UVP7BEBYVKPPQBMN4QOQCJ6Q, seg_pos={0 0}, upload=301248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc): protocol: write tcp 188.166.225.170:40614->104.36.82.44:4898: use of closed network connection; write tcp 188.166.225.170:40614->104.36.82.44:4898: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40614->104.36.82.44:4898: use of closed network connection
Jul 9 01:44:21.573276430 (376h33m56.343917755s) [seg_pos={0 0}, upload=301248]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 01:44:21.573349429 (376h33m56.343990764s) [upload=301248]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 01:44:21.573353537 (376h33m56.343994873s) [upload=301248]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 01:44:21.865577192 (376h33m56.636218527s) [upload=301248]: Done uploading segments...
Jul 9 01:44:21.865594906 (376h33m56.636236232s) [upload=301248]: Upload finished. err: <nil>
Duration: 41.407442992s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
15 out of 60 is failed (threshold=3) [..X.X....X...X.X..X...X..X............X......X..X...X.X.X.X.] duration:
23.453331719s
FAIL (2025-07-09 01:55:55 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 23.45335262s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...................X......................X.................] duration:
25.050984545s
FAIL (2025-07-09 01:55:32 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Jul 9 01:41:35.784016383 (376h31m10.554657718s) [upload=301225]: Starting upload
Jul 9 01:41:35.784073732 (376h31m10.554715067s) [upload=301225]: Done starting upload
Jul 9 01:41:35.784180246 (376h31m10.554821571s) [upload=301225]: Uploading segments...
Jul 9 01:41:35.784275688 (376h31m10.554917023s) [upload=301225]: Got next segment. Inline: false
Jul 9 01:41:35.784309523 (376h31m10.554950848s) [upload=301225]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 01:41:35.784314523 (376h31m10.554955858s) [upload=301225]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 01:41:36.035706020 (376h31m10.806347375s) [seg_pos={0 0}, upload=301225]: Begin upload segment...
Jul 9 01:41:36.035755273 (376h31m10.806396598s) [seg_pos={0 0}, upload=301225]: Done begin upload segment.
Jul 9 01:41:36.035768209 (376h31m10.806409574s) [noise=true , address=45.91.116.14:4190 , piece_id=CZGKNHBIJP7HZIC65I6ESRFRQLBKTEAQJVVUQHHZHFO7TNNROJ4A, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.035782255 (376h31m10.80642359s) [noise=true , address=104.36.82.16:4294 , piece_id=CE4MNKGLLBYQAQ7KGVTTGUMWRTD2NTWF55NOPILQC5YPXDQT3Z6A, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.035859012 (376h31m10.806500367s) [noise=true , address=104.36.82.5:4840 , piece_id=QKGEMZVR5P3B6ARIGS4VQDSWGAOVC5DU65Y3VLXUY2MAYVKDUQCQ, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.036404877 (376h31m10.807046232s) [noise=true , address=45.91.116.8:4045 , piece_id=Z37SMFGUPVJVP6RNGUIDH6Q456OJFVZ7SNFFYT3NT3AOXVYR5TXA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.036420547 (376h31m10.807061892s) [noise=true , address=77.90.31.19:4843 , piece_id=JH6TQLWG4LCPENFHL6ORQQBR5YDUOU2QAY7OLQ5KYXOK2DW5DRGA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.036942675 (376h31m10.80758403s) [noise=true , address=104.36.82.9:4007 , piece_id=TMZJKWZDZHE4ANML45KWGRPDG5UOTNIYUZFDH3LJYZHMU6LBNTIA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.037006067 (376h31m10.807647412s) [noise=true , address=45.91.116.10:4082 , piece_id=DNZJWYESTZZY2QVM2SRD32AS6GDWSSO7ODNWMHBXJAGRNQ3FMSMQ, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.037346698 (376h31m10.807988043s) [noise=true , address=45.91.116.6:4975 , piece_id=LDM5B2S4NP43AXC3DJXTUM66Z6EM3SBKPP6JHTMLA7RZATZQ4QQA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.037776601 (376h31m10.808417936s) [noise=true , address=104.36.82.11:4114 , piece_id=5ISRL5WJST2SCUOPBB7KX53T3PKO2CH5X7QKCPGSKSDO3KA5BYHQ, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.038177858 (376h31m10.808819193s) [noise=true , address=104.36.82.14:4222 , piece_id=FU4BHM26LXD5XUTT333YO7ETORBXWP6QWCLUUJI2MBMPWLEFMGAA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.038258122 (376h31m10.808899467s) [noise=true , address=77.90.31.12:4592 , piece_id=J7LR65VEHL5D7XBO2HMEMFZVFBPEES5IL7FQTV5IJK4ZHEHGDKOA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.038562023 (376h31m10.809203359s) [noise=true , address=45.91.116.13:4153 , piece_id=UMTA7DVZBXEYRQQXBO33BCQL4W5L5RKJAYACZHYJMHCX6XORWSWQ, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.038582503 (376h31m10.809223838s) [noise=true , address=104.36.82.2:4730 , piece_id=EVW56CTEQD7BPUSYS2U5NXO4VRG6YZVEITV6OU5XKT3AU3MRWRYQ, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.038924036 (376h31m10.809565371s) [noise=true , address=104.36.82.8:4969 , piece_id=6IKLPKOPP2HBAOC5FAI5IE5SNW2B2UWNWT3Q6S4WWKGDLRCARVWA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.039254127 (376h31m10.809895453s) [noise=true , address=77.90.31.21:4878 , piece_id=4YNDQENFOKMHFJFVWWBC7UK4TQP745OUW7S2P43BICF2SNFAWIHA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.039603647 (376h31m10.810244982s) [noise=true , address=45.91.116.12:4227 , piece_id=23YQLIAPNPNMSA3LL67PNYCRDMWJ4GXZY3VF354XXUH5YEXFWVUA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:37.680432693 (376h31m12.451074028s) [upload=301225]: Next returned nil segment
Jul 9 01:41:37.680461809 (376h31m12.451103144s) [upload=301225]: Waiting for error group managing segments...
Jul 9 01:41:38.011139865 (376h31m12.7817812s) [noise=true , address=104.36.82.16:4294 , piece_id=CE4MNKGLLBYQAQ7KGVTTGUMWRTD2NTWF55NOPILQC5YPXDQT3Z6A, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.014224286 (376h31m12.784865621s) [noise=true , address=104.36.82.9:4007 , piece_id=TMZJKWZDZHE4ANML45KWGRPDG5UOTNIYUZFDH3LJYZHMU6LBNTIA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.116297430 (376h31m12.886938765s) [noise=true , address=45.91.116.6:4975 , piece_id=LDM5B2S4NP43AXC3DJXTUM66Z6EM3SBKPP6JHTMLA7RZATZQ4QQA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.118333195 (376h31m12.88897453s) [noise=true , address=45.91.116.13:4153 , piece_id=UMTA7DVZBXEYRQQXBO33BCQL4W5L5RKJAYACZHYJMHCX6XORWSWQ, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.130186111 (376h31m12.900827446s) [noise=true , address=45.91.116.8:4045 , piece_id=Z37SMFGUPVJVP6RNGUIDH6Q456OJFVZ7SNFFYT3NT3AOXVYR5TXA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.130610163 (376h31m12.901251498s) [noise=true , address=45.91.116.10:4082 , piece_id=DNZJWYESTZZY2QVM2SRD32AS6GDWSSO7ODNWMHBXJAGRNQ3FMSMQ, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.171163518 (376h31m12.941804854s) [noise=true , address=77.90.31.19:4843 , piece_id=JH6TQLWG4LCPENFHL6ORQQBR5YDUOU2QAY7OLQ5KYXOK2DW5DRGA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.187728911 (376h31m12.958370246s) [noise=true , address=77.90.31.12:4592 , piece_id=J7LR65VEHL5D7XBO2HMEMFZVFBPEES5IL7FQTV5IJK4ZHEHGDKOA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.190933191 (376h31m12.961574526s) [noise=true , address=77.90.31.21:4878 , piece_id=4YNDQENFOKMHFJFVWWBC7UK4TQP745OUW7S2P43BICF2SNFAWIHA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.218675778 (376h31m12.989317113s) [noise=true , address=45.91.116.12:4227 , piece_id=23YQLIAPNPNMSA3LL67PNYCRDMWJ4GXZY3VF354XXUH5YEXFWVUA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.225536314 (376h31m12.996177649s) [noise=true , address=45.91.116.14:4190 , piece_id=CZGKNHBIJP7HZIC65I6ESRFRQLBKTEAQJVVUQHHZHFO7TNNROJ4A, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:42:05.141496679 (376h31m39.912138074s) [noise=true , address=104.36.82.8:4969 , piece_id=6IKLPKOPP2HBAOC5FAI5IE5SNW2B2UWNWT3Q6S4WWKGDLRCARVWA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 01:42:05.141543177 (376h31m39.912184512s) [noise=true , address=104.36.82.14:4222 , piece_id=FU4BHM26LXD5XUTT333YO7ETORBXWP6QWCLUUJI2MBMPWLEFMGAA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 188.166.225.170:60278->104.36.82.14:4222: use of closed network connection; write tcp 188.166.225.170:60278->104.36.82.14:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60278->104.36.82.14:4222: use of closed network connection
Jul 9 01:42:05.141602951 (376h31m39.912244286s) [noise=true , address=104.36.82.5:4840 , piece_id=QKGEMZVR5P3B6ARIGS4VQDSWGAOVC5DU65Y3VLXUY2MAYVKDUQCQ, seg_pos={0 0}, upload=301225]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: write tcp 188.166.225.170:35250->104.36.82.5:4840: use of closed network connection; write tcp 188.166.225.170:35250->104.36.82.5:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35250->104.36.82.5:4840: use of closed network connection
Jul 9 01:42:05.141829285 (376h31m39.91247062s) [noise=true , address=104.36.82.11:4114 , piece_id=5ISRL5WJST2SCUOPBB7KX53T3PKO2CH5X7QKCPGSKSDO3KA5BYHQ, seg_pos={0 0}, upload=301225]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 188.166.225.170:54484->104.36.82.11:4114: use of closed network connection; write tcp 188.166.225.170:54484->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54484->104.36.82.11:4114: use of closed network connection
Jul 9 01:42:05.141852700 (376h31m39.912494025s) [noise=true , address=104.36.82.2:4730 , piece_id=EVW56CTEQD7BPUSYS2U5NXO4VRG6YZVEITV6OU5XKT3AU3MRWRYQ, seg_pos={0 0}, upload=301225]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 188.166.225.170:59748->104.36.82.2:4730: use of closed network connection; write tcp 188.166.225.170:59748->104.36.82.2:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59748->104.36.82.2:4730: use of closed network connection
Jul 9 01:42:05.141879792 (376h31m39.912521127s) [seg_pos={0 0}, upload=301225]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 9 01:42:05.366924712 (376h31m40.137566037s) [upload=301225]: Done uploading segments...
Jul 9 01:42:05.366949890 (376h31m40.137591225s) [upload=301225]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 25.051011135s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
15 out of 60 is failed (threshold=3) [...X.X....X...X.X..X...X..X............X......X..X...X.X.X.X] duration:
30.001155873s
FAIL (2025-07-09 01:55:22 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.00117514s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
16 out of 60 is failed (threshold=3) [................X......XX.XX..X......XXX....XX.....XX...X.XX] duration:
1m0.001375859s
FAIL (2025-07-09 01:55:22 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.001385897s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [..X..........................X......X..........X............] duration:
39.953602091s
FAIL (2025-07-09 01:55:17 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
s) [upload=301248]: Got next segment. Inline: false
Jul 9 01:43:24.322645491 (376h32m59.093286826s) [upload=301248]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 01:43:24.322650441 (376h32m59.093291777s) [upload=301248]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 01:43:24.769774209 (376h32m59.540415544s) [seg_pos={0 0}, upload=301248]: Begin upload segment...
Jul 9 01:43:24.769830437 (376h32m59.540471772s) [seg_pos={0 0}, upload=301248]: Done begin upload segment.
Jul 9 01:43:24.769852048 (376h32m59.540493373s) [noise=true , address=79.127.205.234:4614 , piece_id=JPPMN7KOKXFYF2YDK4RC7AU6KYV2UHQELFEG6ZZ6W5HQ2MKFVN5A, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.769864432 (376h32m59.540505767s) [noise=true , address=45.91.116.11:4123 , piece_id=WSMWN6I2QVBTO6OCRRDCMT3G3TESW3H2ZET5YNWFUPVENPN3UWKQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.770250630 (376h32m59.540891965s) [noise=true , address=104.36.82.22:4511 , piece_id=3AIKVWFRA4CLMPRDYFNX6EA7XO22MB7RQEOVNXPUVUCNFAHPSFCQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.770361543 (376h32m59.541002878s) [noise=true , address=77.90.31.21:4886 , piece_id=YRDZYEV6KU5YISF4DIS3GTZVBFRCZEOIW2DG7VCKHO26NRXMHMVQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.770542389 (376h32m59.541183724s) [noise=true , address=77.90.31.19:4848 , piece_id=RV7TBRZW2YH64X5CLXFHFTJD2WQCEM3P7OW3NYZ2IWC2E3DJDG5A, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.770626239 (376h32m59.541267594s) [noise=true , address=104.36.82.19:4411 , piece_id=R27NIJUXHNTQWK4IXXSVS2QC535JAZJ3K2WV4VANVOGKRYV7YADA, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.770818237 (376h32m59.541459572s) [noise=true , address=104.36.82.40:4955 , piece_id=3TVDS5QR5267W6VKWNXOPW6ZSTXISGXFBIPKK7E3S6ML7C4X4MUQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.770918087 (376h32m59.541559422s) [noise=true , address=104.36.82.12:4160 , piece_id=2Z3FQL5GTT26HC2GPKO5YPOLPB2Z75K4UG3FDPMOEDCCD6Q6MULQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.771127618 (376h32m59.541768953s) [noise=true , address=77.90.31.14:4677 , piece_id=Z7SGXIO722E3U7PWA4AWOUD5T27VSQUCG3ZBVP22WJRFSRKOEXAQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.771146836 (376h32m59.541788171s) [noise=true , address=77.90.31.12:4622 , piece_id=25K7AOPYFLZ6EDAA3SPHHE6S635PN5VB3ZK4LIULBHKQGYYQ3C7A, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.771367858 (376h32m59.542009193s) [noise=true , address=77.90.31.18:4825 , piece_id=NFOYMUTMKMKTPCAEPBTWL43ISOETOAM4WKVAUNHCETNW75IQLLMQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.771421060 (376h32m59.542062395s) [noise=true , address=77.90.31.14:4691 , piece_id=ELMXSLQT6TNOMGNUGM4BEHZ2P7OZWNSX6EOCHDAYOD7OIANFUVHA, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.771586527 (376h32m59.542227852s) [noise=true , address=64.49.13.169:4706 , piece_id=2CKVJCKZJH4F3VC3X7XJOWMYHVHU52POGAA7VMAU4YM6HTVHC5PA, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.771699443 (376h32m59.542340778s) [noise=true , address=104.36.82.44:4898 , piece_id=X47T6NYNODFSVKHJGWIH2WJT7BA3UVP7BEBYVKPPQBMN4QOQCJ6Q, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.771818129 (376h32m59.542459465s) [noise=true , address=104.36.82.34:4551 , piece_id=HWYKZWYEKMJT3JHDPHRWIAFKP5NZQ72OGR4SPR2I2SRRABLA663A, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.772055795 (376h32m59.54269713s) [noise=true , address=77.90.31.10:4543 , piece_id=QCKPGFCSQ3N57BWXMOSW5LAVA4KOI3EVUFU2OMPN7CMAOYAH4RWQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.772089298 (376h32m59.542730623s) [noise=true , address=104.36.82.23:4459 , piece_id=SOG4BGCMVK36OVMKHGIYGSVGX2ZZ2PWZV74S2LQS6IZQRGDYJVYA, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.772274673 (376h32m59.542916008s) [noise=true , address=77.90.31.15:4715 , piece_id=ITPPGCX32ZLU72XXE7RG6AYE3PM7IILVNPXM5A4NJ2S7P6U4Y2FA, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:25.540057386 (376h33m0.310698721s) [noise=true , address=64.49.13.169:4706 , piece_id=2CKVJCKZJH4F3VC3X7XJOWMYHVHU52POGAA7VMAU4YM6HTVHC5PA, seg_pos={0 0}, upload=301248]: Done uploading piece. err: ecclient: upload failed (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7, address:64.49.13.169:4706): protocol: storage node overloaded, request limit: 2000
Jul 9 01:43:25.878055136 (376h33m0.648696481s) [upload=301248]: Next returned nil segment
Jul 9 01:43:25.878073130 (376h33m0.648714465s) [upload=301248]: Waiting for error group managing segments...
Jul 9 01:43:26.175673001 (376h33m0.946314336s) [noise=true , address=79.127.205.234:4614 , piece_id=JPPMN7KOKXFYF2YDK4RC7AU6KYV2UHQELFEG6ZZ6W5HQ2MKFVN5A, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.774276626 (376h33m1.544917961s) [noise=true , address=77.90.31.12:4622 , piece_id=25K7AOPYFLZ6EDAA3SPHHE6S635PN5VB3ZK4LIULBHKQGYYQ3C7A, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.778019557 (376h33m1.548660892s) [noise=true , address=77.90.31.21:4886 , piece_id=YRDZYEV6KU5YISF4DIS3GTZVBFRCZEOIW2DG7VCKHO26NRXMHMVQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.779099603 (376h33m1.549740938s) [noise=true , address=77.90.31.14:4691 , piece_id=ELMXSLQT6TNOMGNUGM4BEHZ2P7OZWNSX6EOCHDAYOD7OIANFUVHA, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.779161091 (376h33m1.549802426s) [noise=true , address=77.90.31.18:4825 , piece_id=NFOYMUTMKMKTPCAEPBTWL43ISOETOAM4WKVAUNHCETNW75IQLLMQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.780045593 (376h33m1.550686928s) [noise=true , address=77.90.31.14:4677 , piece_id=Z7SGXIO722E3U7PWA4AWOUD5T27VSQUCG3ZBVP22WJRFSRKOEXAQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.795740620 (376h33m1.566381976s) [noise=true , address=77.90.31.10:4543 , piece_id=QCKPGFCSQ3N57BWXMOSW5LAVA4KOI3EVUFU2OMPN7CMAOYAH4RWQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.795758073 (376h33m1.566399408s) [noise=true , address=77.90.31.15:4715 , piece_id=ITPPGCX32ZLU72XXE7RG6AYE3PM7IILVNPXM5A4NJ2S7P6U4Y2FA, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.812585147 (376h33m1.583226483s) [noise=true , address=77.90.31.19:4848 , piece_id=RV7TBRZW2YH64X5CLXFHFTJD2WQCEM3P7OW3NYZ2IWC2E3DJDG5A, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:27.386630478 (376h33m2.157271803s) [noise=true , address=45.91.116.11:4123 , piece_id=WSMWN6I2QVBTO6OCRRDCMT3G3TESW3H2ZET5YNWFUPVENPN3UWKQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:37.603914421 (376h33m12.374555756s) [noise=true , address=104.36.82.34:4551 , piece_id=HWYKZWYEKMJT3JHDPHRWIAFKP5NZQ72OGR4SPR2I2SRRABLA663A, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:41.629246242 (376h33m16.399887567s) [noise=true , address=104.36.82.12:4160 , piece_id=2Z3FQL5GTT26HC2GPKO5YPOLPB2Z75K4UG3FDPMOEDCCD6Q6MULQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:44:10.222815170 (376h33m44.993456505s) [noise=true , address=104.36.82.22:4511 , piece_id=3AIKVWFRA4CLMPRDYFNX6EA7XO22MB7RQEOVNXPUVUCNFAHPSFCQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:44:21.573027994 (376h33m56.343669329s) [noise=true , address=104.36.82.23:4459 , piece_id=SOG4BGCMVK36OVMKHGIYGSVGX2ZZ2PWZV74S2LQS6IZQRGDYJVYA, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:44:21.573054295 (376h33m56.34369563s) [seg_pos={0 0}, upload=301248]: Segment reached optimal threshold of 14 pieces.
Jul 9 01:44:21.573140269 (376h33m56.343781604s) [noise=true , address=104.36.82.40:4955 , piece_id=3TVDS5QR5267W6VKWNXOPW6ZSTXISGXFBIPKK7E3S6ML7C4X4MUQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ssLq5bo7og4z8ZPd28f2H6WUExsWohBBZUnciGaH5KKpcQ25d): protocol: expected piece hash; context canceled; EOF
Jul 9 01:44:21.573153795 (376h33m56.34379513s) [noise=true , address=104.36.82.19:4411 , piece_id=R27NIJUXHNTQWK4IXXSVS2QC535JAZJ3K2WV4VANVOGKRYV7YADA, seg_pos={0 0}, upload=301248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nradx4SMYLkuJw3Ad6o9Sq1iNtQR6GDu8qg7tCzZzNcTjVfyd): protocol: expected piece hash; context canceled; EOF
Jul 9 01:44:21.573259487 (376h33m56.343900822s) [noise=true , address=104.36.82.44:4898 , piece_id=X47T6NYNODFSVKHJGWIH2WJT7BA3UVP7BEBYVKPPQBMN4QOQCJ6Q, seg_pos={0 0}, upload=301248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc): protocol: write tcp 188.166.225.170:40614->104.36.82.44:4898: use of closed network connection; write tcp 188.166.225.170:40614->104.36.82.44:4898: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40614->104.36.82.44:4898: use of closed network connection
Jul 9 01:44:21.573276430 (376h33m56.343917755s) [seg_pos={0 0}, upload=301248]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 01:44:21.573349429 (376h33m56.343990764s) [upload=301248]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 01:44:21.573353537 (376h33m56.343994873s) [upload=301248]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 01:44:21.865577192 (376h33m56.636218527s) [upload=301248]: Done uploading segments...
Jul 9 01:44:21.865594906 (376h33m56.636236232s) [upload=301248]: Upload finished. err: <nil>
Duration: 39.953617201s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [....................X......................X................] duration:
5.740406314s
FAIL (2025-07-09 01:54:44 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Jul 9 01:41:35.784016383 (376h31m10.554657718s) [upload=301225]: Starting upload
Jul 9 01:41:35.784073732 (376h31m10.554715067s) [upload=301225]: Done starting upload
Jul 9 01:41:35.784180246 (376h31m10.554821571s) [upload=301225]: Uploading segments...
Jul 9 01:41:35.784275688 (376h31m10.554917023s) [upload=301225]: Got next segment. Inline: false
Jul 9 01:41:35.784309523 (376h31m10.554950848s) [upload=301225]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 01:41:35.784314523 (376h31m10.554955858s) [upload=301225]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 01:41:36.035706020 (376h31m10.806347375s) [seg_pos={0 0}, upload=301225]: Begin upload segment...
Jul 9 01:41:36.035755273 (376h31m10.806396598s) [seg_pos={0 0}, upload=301225]: Done begin upload segment.
Jul 9 01:41:36.035768209 (376h31m10.806409574s) [noise=true , address=45.91.116.14:4190 , piece_id=CZGKNHBIJP7HZIC65I6ESRFRQLBKTEAQJVVUQHHZHFO7TNNROJ4A, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.035782255 (376h31m10.80642359s) [noise=true , address=104.36.82.16:4294 , piece_id=CE4MNKGLLBYQAQ7KGVTTGUMWRTD2NTWF55NOPILQC5YPXDQT3Z6A, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.035859012 (376h31m10.806500367s) [noise=true , address=104.36.82.5:4840 , piece_id=QKGEMZVR5P3B6ARIGS4VQDSWGAOVC5DU65Y3VLXUY2MAYVKDUQCQ, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.036404877 (376h31m10.807046232s) [noise=true , address=45.91.116.8:4045 , piece_id=Z37SMFGUPVJVP6RNGUIDH6Q456OJFVZ7SNFFYT3NT3AOXVYR5TXA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.036420547 (376h31m10.807061892s) [noise=true , address=77.90.31.19:4843 , piece_id=JH6TQLWG4LCPENFHL6ORQQBR5YDUOU2QAY7OLQ5KYXOK2DW5DRGA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.036942675 (376h31m10.80758403s) [noise=true , address=104.36.82.9:4007 , piece_id=TMZJKWZDZHE4ANML45KWGRPDG5UOTNIYUZFDH3LJYZHMU6LBNTIA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.037006067 (376h31m10.807647412s) [noise=true , address=45.91.116.10:4082 , piece_id=DNZJWYESTZZY2QVM2SRD32AS6GDWSSO7ODNWMHBXJAGRNQ3FMSMQ, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.037346698 (376h31m10.807988043s) [noise=true , address=45.91.116.6:4975 , piece_id=LDM5B2S4NP43AXC3DJXTUM66Z6EM3SBKPP6JHTMLA7RZATZQ4QQA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.037776601 (376h31m10.808417936s) [noise=true , address=104.36.82.11:4114 , piece_id=5ISRL5WJST2SCUOPBB7KX53T3PKO2CH5X7QKCPGSKSDO3KA5BYHQ, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.038177858 (376h31m10.808819193s) [noise=true , address=104.36.82.14:4222 , piece_id=FU4BHM26LXD5XUTT333YO7ETORBXWP6QWCLUUJI2MBMPWLEFMGAA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.038258122 (376h31m10.808899467s) [noise=true , address=77.90.31.12:4592 , piece_id=J7LR65VEHL5D7XBO2HMEMFZVFBPEES5IL7FQTV5IJK4ZHEHGDKOA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.038562023 (376h31m10.809203359s) [noise=true , address=45.91.116.13:4153 , piece_id=UMTA7DVZBXEYRQQXBO33BCQL4W5L5RKJAYACZHYJMHCX6XORWSWQ, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.038582503 (376h31m10.809223838s) [noise=true , address=104.36.82.2:4730 , piece_id=EVW56CTEQD7BPUSYS2U5NXO4VRG6YZVEITV6OU5XKT3AU3MRWRYQ, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.038924036 (376h31m10.809565371s) [noise=true , address=104.36.82.8:4969 , piece_id=6IKLPKOPP2HBAOC5FAI5IE5SNW2B2UWNWT3Q6S4WWKGDLRCARVWA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.039254127 (376h31m10.809895453s) [noise=true , address=77.90.31.21:4878 , piece_id=4YNDQENFOKMHFJFVWWBC7UK4TQP745OUW7S2P43BICF2SNFAWIHA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.039603647 (376h31m10.810244982s) [noise=true , address=45.91.116.12:4227 , piece_id=23YQLIAPNPNMSA3LL67PNYCRDMWJ4GXZY3VF354XXUH5YEXFWVUA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:37.680432693 (376h31m12.451074028s) [upload=301225]: Next returned nil segment
Jul 9 01:41:37.680461809 (376h31m12.451103144s) [upload=301225]: Waiting for error group managing segments...
Jul 9 01:41:38.011139865 (376h31m12.7817812s) [noise=true , address=104.36.82.16:4294 , piece_id=CE4MNKGLLBYQAQ7KGVTTGUMWRTD2NTWF55NOPILQC5YPXDQT3Z6A, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.014224286 (376h31m12.784865621s) [noise=true , address=104.36.82.9:4007 , piece_id=TMZJKWZDZHE4ANML45KWGRPDG5UOTNIYUZFDH3LJYZHMU6LBNTIA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.116297430 (376h31m12.886938765s) [noise=true , address=45.91.116.6:4975 , piece_id=LDM5B2S4NP43AXC3DJXTUM66Z6EM3SBKPP6JHTMLA7RZATZQ4QQA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.118333195 (376h31m12.88897453s) [noise=true , address=45.91.116.13:4153 , piece_id=UMTA7DVZBXEYRQQXBO33BCQL4W5L5RKJAYACZHYJMHCX6XORWSWQ, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.130186111 (376h31m12.900827446s) [noise=true , address=45.91.116.8:4045 , piece_id=Z37SMFGUPVJVP6RNGUIDH6Q456OJFVZ7SNFFYT3NT3AOXVYR5TXA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.130610163 (376h31m12.901251498s) [noise=true , address=45.91.116.10:4082 , piece_id=DNZJWYESTZZY2QVM2SRD32AS6GDWSSO7ODNWMHBXJAGRNQ3FMSMQ, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.171163518 (376h31m12.941804854s) [noise=true , address=77.90.31.19:4843 , piece_id=JH6TQLWG4LCPENFHL6ORQQBR5YDUOU2QAY7OLQ5KYXOK2DW5DRGA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.187728911 (376h31m12.958370246s) [noise=true , address=77.90.31.12:4592 , piece_id=J7LR65VEHL5D7XBO2HMEMFZVFBPEES5IL7FQTV5IJK4ZHEHGDKOA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.190933191 (376h31m12.961574526s) [noise=true , address=77.90.31.21:4878 , piece_id=4YNDQENFOKMHFJFVWWBC7UK4TQP745OUW7S2P43BICF2SNFAWIHA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.218675778 (376h31m12.989317113s) [noise=true , address=45.91.116.12:4227 , piece_id=23YQLIAPNPNMSA3LL67PNYCRDMWJ4GXZY3VF354XXUH5YEXFWVUA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.225536314 (376h31m12.996177649s) [noise=true , address=45.91.116.14:4190 , piece_id=CZGKNHBIJP7HZIC65I6ESRFRQLBKTEAQJVVUQHHZHFO7TNNROJ4A, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:42:05.141496679 (376h31m39.912138074s) [noise=true , address=104.36.82.8:4969 , piece_id=6IKLPKOPP2HBAOC5FAI5IE5SNW2B2UWNWT3Q6S4WWKGDLRCARVWA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 01:42:05.141543177 (376h31m39.912184512s) [noise=true , address=104.36.82.14:4222 , piece_id=FU4BHM26LXD5XUTT333YO7ETORBXWP6QWCLUUJI2MBMPWLEFMGAA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 188.166.225.170:60278->104.36.82.14:4222: use of closed network connection; write tcp 188.166.225.170:60278->104.36.82.14:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60278->104.36.82.14:4222: use of closed network connection
Jul 9 01:42:05.141602951 (376h31m39.912244286s) [noise=true , address=104.36.82.5:4840 , piece_id=QKGEMZVR5P3B6ARIGS4VQDSWGAOVC5DU65Y3VLXUY2MAYVKDUQCQ, seg_pos={0 0}, upload=301225]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: write tcp 188.166.225.170:35250->104.36.82.5:4840: use of closed network connection; write tcp 188.166.225.170:35250->104.36.82.5:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35250->104.36.82.5:4840: use of closed network connection
Jul 9 01:42:05.141829285 (376h31m39.91247062s) [noise=true , address=104.36.82.11:4114 , piece_id=5ISRL5WJST2SCUOPBB7KX53T3PKO2CH5X7QKCPGSKSDO3KA5BYHQ, seg_pos={0 0}, upload=301225]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 188.166.225.170:54484->104.36.82.11:4114: use of closed network connection; write tcp 188.166.225.170:54484->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54484->104.36.82.11:4114: use of closed network connection
Jul 9 01:42:05.141852700 (376h31m39.912494025s) [noise=true , address=104.36.82.2:4730 , piece_id=EVW56CTEQD7BPUSYS2U5NXO4VRG6YZVEITV6OU5XKT3AU3MRWRYQ, seg_pos={0 0}, upload=301225]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 188.166.225.170:59748->104.36.82.2:4730: use of closed network connection; write tcp 188.166.225.170:59748->104.36.82.2:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59748->104.36.82.2:4730: use of closed network connection
Jul 9 01:42:05.141879792 (376h31m39.912521127s) [seg_pos={0 0}, upload=301225]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 9 01:42:05.366924712 (376h31m40.137566037s) [upload=301225]: Done uploading segments...
Jul 9 01:42:05.366949890 (376h31m40.137591225s) [upload=301225]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 5.740430411s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
15 out of 60 is failed (threshold=3) [X...X.X....X...X.X..X...X..X............X......X..X...X.X.X.] duration:
8.58082556s
FAIL (2025-07-09 01:54:43 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 8.580843334s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [...X..........................X......X..........X...........] duration:
25.708693881s
FAIL (2025-07-09 01:54:37 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
s) [upload=301248]: Got next segment. Inline: false
Jul 9 01:43:24.322645491 (376h32m59.093286826s) [upload=301248]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 01:43:24.322650441 (376h32m59.093291777s) [upload=301248]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 01:43:24.769774209 (376h32m59.540415544s) [seg_pos={0 0}, upload=301248]: Begin upload segment...
Jul 9 01:43:24.769830437 (376h32m59.540471772s) [seg_pos={0 0}, upload=301248]: Done begin upload segment.
Jul 9 01:43:24.769852048 (376h32m59.540493373s) [noise=true , address=79.127.205.234:4614 , piece_id=JPPMN7KOKXFYF2YDK4RC7AU6KYV2UHQELFEG6ZZ6W5HQ2MKFVN5A, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.769864432 (376h32m59.540505767s) [noise=true , address=45.91.116.11:4123 , piece_id=WSMWN6I2QVBTO6OCRRDCMT3G3TESW3H2ZET5YNWFUPVENPN3UWKQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.770250630 (376h32m59.540891965s) [noise=true , address=104.36.82.22:4511 , piece_id=3AIKVWFRA4CLMPRDYFNX6EA7XO22MB7RQEOVNXPUVUCNFAHPSFCQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.770361543 (376h32m59.541002878s) [noise=true , address=77.90.31.21:4886 , piece_id=YRDZYEV6KU5YISF4DIS3GTZVBFRCZEOIW2DG7VCKHO26NRXMHMVQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.770542389 (376h32m59.541183724s) [noise=true , address=77.90.31.19:4848 , piece_id=RV7TBRZW2YH64X5CLXFHFTJD2WQCEM3P7OW3NYZ2IWC2E3DJDG5A, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.770626239 (376h32m59.541267594s) [noise=true , address=104.36.82.19:4411 , piece_id=R27NIJUXHNTQWK4IXXSVS2QC535JAZJ3K2WV4VANVOGKRYV7YADA, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.770818237 (376h32m59.541459572s) [noise=true , address=104.36.82.40:4955 , piece_id=3TVDS5QR5267W6VKWNXOPW6ZSTXISGXFBIPKK7E3S6ML7C4X4MUQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.770918087 (376h32m59.541559422s) [noise=true , address=104.36.82.12:4160 , piece_id=2Z3FQL5GTT26HC2GPKO5YPOLPB2Z75K4UG3FDPMOEDCCD6Q6MULQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.771127618 (376h32m59.541768953s) [noise=true , address=77.90.31.14:4677 , piece_id=Z7SGXIO722E3U7PWA4AWOUD5T27VSQUCG3ZBVP22WJRFSRKOEXAQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.771146836 (376h32m59.541788171s) [noise=true , address=77.90.31.12:4622 , piece_id=25K7AOPYFLZ6EDAA3SPHHE6S635PN5VB3ZK4LIULBHKQGYYQ3C7A, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.771367858 (376h32m59.542009193s) [noise=true , address=77.90.31.18:4825 , piece_id=NFOYMUTMKMKTPCAEPBTWL43ISOETOAM4WKVAUNHCETNW75IQLLMQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.771421060 (376h32m59.542062395s) [noise=true , address=77.90.31.14:4691 , piece_id=ELMXSLQT6TNOMGNUGM4BEHZ2P7OZWNSX6EOCHDAYOD7OIANFUVHA, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.771586527 (376h32m59.542227852s) [noise=true , address=64.49.13.169:4706 , piece_id=2CKVJCKZJH4F3VC3X7XJOWMYHVHU52POGAA7VMAU4YM6HTVHC5PA, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.771699443 (376h32m59.542340778s) [noise=true , address=104.36.82.44:4898 , piece_id=X47T6NYNODFSVKHJGWIH2WJT7BA3UVP7BEBYVKPPQBMN4QOQCJ6Q, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.771818129 (376h32m59.542459465s) [noise=true , address=104.36.82.34:4551 , piece_id=HWYKZWYEKMJT3JHDPHRWIAFKP5NZQ72OGR4SPR2I2SRRABLA663A, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.772055795 (376h32m59.54269713s) [noise=true , address=77.90.31.10:4543 , piece_id=QCKPGFCSQ3N57BWXMOSW5LAVA4KOI3EVUFU2OMPN7CMAOYAH4RWQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.772089298 (376h32m59.542730623s) [noise=true , address=104.36.82.23:4459 , piece_id=SOG4BGCMVK36OVMKHGIYGSVGX2ZZ2PWZV74S2LQS6IZQRGDYJVYA, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.772274673 (376h32m59.542916008s) [noise=true , address=77.90.31.15:4715 , piece_id=ITPPGCX32ZLU72XXE7RG6AYE3PM7IILVNPXM5A4NJ2S7P6U4Y2FA, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:25.540057386 (376h33m0.310698721s) [noise=true , address=64.49.13.169:4706 , piece_id=2CKVJCKZJH4F3VC3X7XJOWMYHVHU52POGAA7VMAU4YM6HTVHC5PA, seg_pos={0 0}, upload=301248]: Done uploading piece. err: ecclient: upload failed (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7, address:64.49.13.169:4706): protocol: storage node overloaded, request limit: 2000
Jul 9 01:43:25.878055136 (376h33m0.648696481s) [upload=301248]: Next returned nil segment
Jul 9 01:43:25.878073130 (376h33m0.648714465s) [upload=301248]: Waiting for error group managing segments...
Jul 9 01:43:26.175673001 (376h33m0.946314336s) [noise=true , address=79.127.205.234:4614 , piece_id=JPPMN7KOKXFYF2YDK4RC7AU6KYV2UHQELFEG6ZZ6W5HQ2MKFVN5A, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.774276626 (376h33m1.544917961s) [noise=true , address=77.90.31.12:4622 , piece_id=25K7AOPYFLZ6EDAA3SPHHE6S635PN5VB3ZK4LIULBHKQGYYQ3C7A, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.778019557 (376h33m1.548660892s) [noise=true , address=77.90.31.21:4886 , piece_id=YRDZYEV6KU5YISF4DIS3GTZVBFRCZEOIW2DG7VCKHO26NRXMHMVQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.779099603 (376h33m1.549740938s) [noise=true , address=77.90.31.14:4691 , piece_id=ELMXSLQT6TNOMGNUGM4BEHZ2P7OZWNSX6EOCHDAYOD7OIANFUVHA, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.779161091 (376h33m1.549802426s) [noise=true , address=77.90.31.18:4825 , piece_id=NFOYMUTMKMKTPCAEPBTWL43ISOETOAM4WKVAUNHCETNW75IQLLMQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.780045593 (376h33m1.550686928s) [noise=true , address=77.90.31.14:4677 , piece_id=Z7SGXIO722E3U7PWA4AWOUD5T27VSQUCG3ZBVP22WJRFSRKOEXAQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.795740620 (376h33m1.566381976s) [noise=true , address=77.90.31.10:4543 , piece_id=QCKPGFCSQ3N57BWXMOSW5LAVA4KOI3EVUFU2OMPN7CMAOYAH4RWQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.795758073 (376h33m1.566399408s) [noise=true , address=77.90.31.15:4715 , piece_id=ITPPGCX32ZLU72XXE7RG6AYE3PM7IILVNPXM5A4NJ2S7P6U4Y2FA, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.812585147 (376h33m1.583226483s) [noise=true , address=77.90.31.19:4848 , piece_id=RV7TBRZW2YH64X5CLXFHFTJD2WQCEM3P7OW3NYZ2IWC2E3DJDG5A, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:27.386630478 (376h33m2.157271803s) [noise=true , address=45.91.116.11:4123 , piece_id=WSMWN6I2QVBTO6OCRRDCMT3G3TESW3H2ZET5YNWFUPVENPN3UWKQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:37.603914421 (376h33m12.374555756s) [noise=true , address=104.36.82.34:4551 , piece_id=HWYKZWYEKMJT3JHDPHRWIAFKP5NZQ72OGR4SPR2I2SRRABLA663A, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:41.629246242 (376h33m16.399887567s) [noise=true , address=104.36.82.12:4160 , piece_id=2Z3FQL5GTT26HC2GPKO5YPOLPB2Z75K4UG3FDPMOEDCCD6Q6MULQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:44:10.222815170 (376h33m44.993456505s) [noise=true , address=104.36.82.22:4511 , piece_id=3AIKVWFRA4CLMPRDYFNX6EA7XO22MB7RQEOVNXPUVUCNFAHPSFCQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:44:21.573027994 (376h33m56.343669329s) [noise=true , address=104.36.82.23:4459 , piece_id=SOG4BGCMVK36OVMKHGIYGSVGX2ZZ2PWZV74S2LQS6IZQRGDYJVYA, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:44:21.573054295 (376h33m56.34369563s) [seg_pos={0 0}, upload=301248]: Segment reached optimal threshold of 14 pieces.
Jul 9 01:44:21.573140269 (376h33m56.343781604s) [noise=true , address=104.36.82.40:4955 , piece_id=3TVDS5QR5267W6VKWNXOPW6ZSTXISGXFBIPKK7E3S6ML7C4X4MUQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ssLq5bo7og4z8ZPd28f2H6WUExsWohBBZUnciGaH5KKpcQ25d): protocol: expected piece hash; context canceled; EOF
Jul 9 01:44:21.573153795 (376h33m56.34379513s) [noise=true , address=104.36.82.19:4411 , piece_id=R27NIJUXHNTQWK4IXXSVS2QC535JAZJ3K2WV4VANVOGKRYV7YADA, seg_pos={0 0}, upload=301248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nradx4SMYLkuJw3Ad6o9Sq1iNtQR6GDu8qg7tCzZzNcTjVfyd): protocol: expected piece hash; context canceled; EOF
Jul 9 01:44:21.573259487 (376h33m56.343900822s) [noise=true , address=104.36.82.44:4898 , piece_id=X47T6NYNODFSVKHJGWIH2WJT7BA3UVP7BEBYVKPPQBMN4QOQCJ6Q, seg_pos={0 0}, upload=301248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc): protocol: write tcp 188.166.225.170:40614->104.36.82.44:4898: use of closed network connection; write tcp 188.166.225.170:40614->104.36.82.44:4898: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40614->104.36.82.44:4898: use of closed network connection
Jul 9 01:44:21.573276430 (376h33m56.343917755s) [seg_pos={0 0}, upload=301248]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 01:44:21.573349429 (376h33m56.343990764s) [upload=301248]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 01:44:21.573353537 (376h33m56.343994873s) [upload=301248]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 01:44:21.865577192 (376h33m56.636218527s) [upload=301248]: Done uploading segments...
Jul 9 01:44:21.865594906 (376h33m56.636236232s) [upload=301248]: Upload finished. err: <nil>
Duration: 25.708718329s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
15 out of 60 is failed (threshold=3) [.X...X.X....X...X.X..X...X..X............X......X..X...X.X.X] duration:
30.0004767s
FAIL (2025-07-09 01:54:25 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000498543s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
15 out of 60 is failed (threshold=3) [.................X......XX.XX..X......XXX....XX.....XX...X.X] duration:
1m0.000765591s
FAIL (2025-07-09 01:54:21 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.000782433s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.....................X......................X...............] duration:
5.808898593s
FAIL (2025-07-09 01:54:20 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Jul 9 01:41:35.784016383 (376h31m10.554657718s) [upload=301225]: Starting upload
Jul 9 01:41:35.784073732 (376h31m10.554715067s) [upload=301225]: Done starting upload
Jul 9 01:41:35.784180246 (376h31m10.554821571s) [upload=301225]: Uploading segments...
Jul 9 01:41:35.784275688 (376h31m10.554917023s) [upload=301225]: Got next segment. Inline: false
Jul 9 01:41:35.784309523 (376h31m10.554950848s) [upload=301225]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 01:41:35.784314523 (376h31m10.554955858s) [upload=301225]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 01:41:36.035706020 (376h31m10.806347375s) [seg_pos={0 0}, upload=301225]: Begin upload segment...
Jul 9 01:41:36.035755273 (376h31m10.806396598s) [seg_pos={0 0}, upload=301225]: Done begin upload segment.
Jul 9 01:41:36.035768209 (376h31m10.806409574s) [noise=true , address=45.91.116.14:4190 , piece_id=CZGKNHBIJP7HZIC65I6ESRFRQLBKTEAQJVVUQHHZHFO7TNNROJ4A, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.035782255 (376h31m10.80642359s) [noise=true , address=104.36.82.16:4294 , piece_id=CE4MNKGLLBYQAQ7KGVTTGUMWRTD2NTWF55NOPILQC5YPXDQT3Z6A, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.035859012 (376h31m10.806500367s) [noise=true , address=104.36.82.5:4840 , piece_id=QKGEMZVR5P3B6ARIGS4VQDSWGAOVC5DU65Y3VLXUY2MAYVKDUQCQ, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.036404877 (376h31m10.807046232s) [noise=true , address=45.91.116.8:4045 , piece_id=Z37SMFGUPVJVP6RNGUIDH6Q456OJFVZ7SNFFYT3NT3AOXVYR5TXA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.036420547 (376h31m10.807061892s) [noise=true , address=77.90.31.19:4843 , piece_id=JH6TQLWG4LCPENFHL6ORQQBR5YDUOU2QAY7OLQ5KYXOK2DW5DRGA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.036942675 (376h31m10.80758403s) [noise=true , address=104.36.82.9:4007 , piece_id=TMZJKWZDZHE4ANML45KWGRPDG5UOTNIYUZFDH3LJYZHMU6LBNTIA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.037006067 (376h31m10.807647412s) [noise=true , address=45.91.116.10:4082 , piece_id=DNZJWYESTZZY2QVM2SRD32AS6GDWSSO7ODNWMHBXJAGRNQ3FMSMQ, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.037346698 (376h31m10.807988043s) [noise=true , address=45.91.116.6:4975 , piece_id=LDM5B2S4NP43AXC3DJXTUM66Z6EM3SBKPP6JHTMLA7RZATZQ4QQA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.037776601 (376h31m10.808417936s) [noise=true , address=104.36.82.11:4114 , piece_id=5ISRL5WJST2SCUOPBB7KX53T3PKO2CH5X7QKCPGSKSDO3KA5BYHQ, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.038177858 (376h31m10.808819193s) [noise=true , address=104.36.82.14:4222 , piece_id=FU4BHM26LXD5XUTT333YO7ETORBXWP6QWCLUUJI2MBMPWLEFMGAA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.038258122 (376h31m10.808899467s) [noise=true , address=77.90.31.12:4592 , piece_id=J7LR65VEHL5D7XBO2HMEMFZVFBPEES5IL7FQTV5IJK4ZHEHGDKOA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.038562023 (376h31m10.809203359s) [noise=true , address=45.91.116.13:4153 , piece_id=UMTA7DVZBXEYRQQXBO33BCQL4W5L5RKJAYACZHYJMHCX6XORWSWQ, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.038582503 (376h31m10.809223838s) [noise=true , address=104.36.82.2:4730 , piece_id=EVW56CTEQD7BPUSYS2U5NXO4VRG6YZVEITV6OU5XKT3AU3MRWRYQ, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.038924036 (376h31m10.809565371s) [noise=true , address=104.36.82.8:4969 , piece_id=6IKLPKOPP2HBAOC5FAI5IE5SNW2B2UWNWT3Q6S4WWKGDLRCARVWA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.039254127 (376h31m10.809895453s) [noise=true , address=77.90.31.21:4878 , piece_id=4YNDQENFOKMHFJFVWWBC7UK4TQP745OUW7S2P43BICF2SNFAWIHA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.039603647 (376h31m10.810244982s) [noise=true , address=45.91.116.12:4227 , piece_id=23YQLIAPNPNMSA3LL67PNYCRDMWJ4GXZY3VF354XXUH5YEXFWVUA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:37.680432693 (376h31m12.451074028s) [upload=301225]: Next returned nil segment
Jul 9 01:41:37.680461809 (376h31m12.451103144s) [upload=301225]: Waiting for error group managing segments...
Jul 9 01:41:38.011139865 (376h31m12.7817812s) [noise=true , address=104.36.82.16:4294 , piece_id=CE4MNKGLLBYQAQ7KGVTTGUMWRTD2NTWF55NOPILQC5YPXDQT3Z6A, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.014224286 (376h31m12.784865621s) [noise=true , address=104.36.82.9:4007 , piece_id=TMZJKWZDZHE4ANML45KWGRPDG5UOTNIYUZFDH3LJYZHMU6LBNTIA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.116297430 (376h31m12.886938765s) [noise=true , address=45.91.116.6:4975 , piece_id=LDM5B2S4NP43AXC3DJXTUM66Z6EM3SBKPP6JHTMLA7RZATZQ4QQA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.118333195 (376h31m12.88897453s) [noise=true , address=45.91.116.13:4153 , piece_id=UMTA7DVZBXEYRQQXBO33BCQL4W5L5RKJAYACZHYJMHCX6XORWSWQ, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.130186111 (376h31m12.900827446s) [noise=true , address=45.91.116.8:4045 , piece_id=Z37SMFGUPVJVP6RNGUIDH6Q456OJFVZ7SNFFYT3NT3AOXVYR5TXA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.130610163 (376h31m12.901251498s) [noise=true , address=45.91.116.10:4082 , piece_id=DNZJWYESTZZY2QVM2SRD32AS6GDWSSO7ODNWMHBXJAGRNQ3FMSMQ, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.171163518 (376h31m12.941804854s) [noise=true , address=77.90.31.19:4843 , piece_id=JH6TQLWG4LCPENFHL6ORQQBR5YDUOU2QAY7OLQ5KYXOK2DW5DRGA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.187728911 (376h31m12.958370246s) [noise=true , address=77.90.31.12:4592 , piece_id=J7LR65VEHL5D7XBO2HMEMFZVFBPEES5IL7FQTV5IJK4ZHEHGDKOA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.190933191 (376h31m12.961574526s) [noise=true , address=77.90.31.21:4878 , piece_id=4YNDQENFOKMHFJFVWWBC7UK4TQP745OUW7S2P43BICF2SNFAWIHA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.218675778 (376h31m12.989317113s) [noise=true , address=45.91.116.12:4227 , piece_id=23YQLIAPNPNMSA3LL67PNYCRDMWJ4GXZY3VF354XXUH5YEXFWVUA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.225536314 (376h31m12.996177649s) [noise=true , address=45.91.116.14:4190 , piece_id=CZGKNHBIJP7HZIC65I6ESRFRQLBKTEAQJVVUQHHZHFO7TNNROJ4A, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:42:05.141496679 (376h31m39.912138074s) [noise=true , address=104.36.82.8:4969 , piece_id=6IKLPKOPP2HBAOC5FAI5IE5SNW2B2UWNWT3Q6S4WWKGDLRCARVWA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 01:42:05.141543177 (376h31m39.912184512s) [noise=true , address=104.36.82.14:4222 , piece_id=FU4BHM26LXD5XUTT333YO7ETORBXWP6QWCLUUJI2MBMPWLEFMGAA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 188.166.225.170:60278->104.36.82.14:4222: use of closed network connection; write tcp 188.166.225.170:60278->104.36.82.14:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60278->104.36.82.14:4222: use of closed network connection
Jul 9 01:42:05.141602951 (376h31m39.912244286s) [noise=true , address=104.36.82.5:4840 , piece_id=QKGEMZVR5P3B6ARIGS4VQDSWGAOVC5DU65Y3VLXUY2MAYVKDUQCQ, seg_pos={0 0}, upload=301225]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: write tcp 188.166.225.170:35250->104.36.82.5:4840: use of closed network connection; write tcp 188.166.225.170:35250->104.36.82.5:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35250->104.36.82.5:4840: use of closed network connection
Jul 9 01:42:05.141829285 (376h31m39.91247062s) [noise=true , address=104.36.82.11:4114 , piece_id=5ISRL5WJST2SCUOPBB7KX53T3PKO2CH5X7QKCPGSKSDO3KA5BYHQ, seg_pos={0 0}, upload=301225]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 188.166.225.170:54484->104.36.82.11:4114: use of closed network connection; write tcp 188.166.225.170:54484->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54484->104.36.82.11:4114: use of closed network connection
Jul 9 01:42:05.141852700 (376h31m39.912494025s) [noise=true , address=104.36.82.2:4730 , piece_id=EVW56CTEQD7BPUSYS2U5NXO4VRG6YZVEITV6OU5XKT3AU3MRWRYQ, seg_pos={0 0}, upload=301225]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 188.166.225.170:59748->104.36.82.2:4730: use of closed network connection; write tcp 188.166.225.170:59748->104.36.82.2:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59748->104.36.82.2:4730: use of closed network connection
Jul 9 01:42:05.141879792 (376h31m39.912521127s) [seg_pos={0 0}, upload=301225]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 9 01:42:05.366924712 (376h31m40.137566037s) [upload=301225]: Done uploading segments...
Jul 9 01:42:05.366949890 (376h31m40.137591225s) [upload=301225]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 5.808924112s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [....X..........................X......X..........X..........] duration:
28.133204627s
FAIL (2025-07-09 01:53:49 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
s) [upload=301248]: Got next segment. Inline: false
Jul 9 01:43:24.322645491 (376h32m59.093286826s) [upload=301248]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 01:43:24.322650441 (376h32m59.093291777s) [upload=301248]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 01:43:24.769774209 (376h32m59.540415544s) [seg_pos={0 0}, upload=301248]: Begin upload segment...
Jul 9 01:43:24.769830437 (376h32m59.540471772s) [seg_pos={0 0}, upload=301248]: Done begin upload segment.
Jul 9 01:43:24.769852048 (376h32m59.540493373s) [noise=true , address=79.127.205.234:4614 , piece_id=JPPMN7KOKXFYF2YDK4RC7AU6KYV2UHQELFEG6ZZ6W5HQ2MKFVN5A, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.769864432 (376h32m59.540505767s) [noise=true , address=45.91.116.11:4123 , piece_id=WSMWN6I2QVBTO6OCRRDCMT3G3TESW3H2ZET5YNWFUPVENPN3UWKQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.770250630 (376h32m59.540891965s) [noise=true , address=104.36.82.22:4511 , piece_id=3AIKVWFRA4CLMPRDYFNX6EA7XO22MB7RQEOVNXPUVUCNFAHPSFCQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.770361543 (376h32m59.541002878s) [noise=true , address=77.90.31.21:4886 , piece_id=YRDZYEV6KU5YISF4DIS3GTZVBFRCZEOIW2DG7VCKHO26NRXMHMVQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.770542389 (376h32m59.541183724s) [noise=true , address=77.90.31.19:4848 , piece_id=RV7TBRZW2YH64X5CLXFHFTJD2WQCEM3P7OW3NYZ2IWC2E3DJDG5A, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.770626239 (376h32m59.541267594s) [noise=true , address=104.36.82.19:4411 , piece_id=R27NIJUXHNTQWK4IXXSVS2QC535JAZJ3K2WV4VANVOGKRYV7YADA, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.770818237 (376h32m59.541459572s) [noise=true , address=104.36.82.40:4955 , piece_id=3TVDS5QR5267W6VKWNXOPW6ZSTXISGXFBIPKK7E3S6ML7C4X4MUQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.770918087 (376h32m59.541559422s) [noise=true , address=104.36.82.12:4160 , piece_id=2Z3FQL5GTT26HC2GPKO5YPOLPB2Z75K4UG3FDPMOEDCCD6Q6MULQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.771127618 (376h32m59.541768953s) [noise=true , address=77.90.31.14:4677 , piece_id=Z7SGXIO722E3U7PWA4AWOUD5T27VSQUCG3ZBVP22WJRFSRKOEXAQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.771146836 (376h32m59.541788171s) [noise=true , address=77.90.31.12:4622 , piece_id=25K7AOPYFLZ6EDAA3SPHHE6S635PN5VB3ZK4LIULBHKQGYYQ3C7A, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.771367858 (376h32m59.542009193s) [noise=true , address=77.90.31.18:4825 , piece_id=NFOYMUTMKMKTPCAEPBTWL43ISOETOAM4WKVAUNHCETNW75IQLLMQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.771421060 (376h32m59.542062395s) [noise=true , address=77.90.31.14:4691 , piece_id=ELMXSLQT6TNOMGNUGM4BEHZ2P7OZWNSX6EOCHDAYOD7OIANFUVHA, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.771586527 (376h32m59.542227852s) [noise=true , address=64.49.13.169:4706 , piece_id=2CKVJCKZJH4F3VC3X7XJOWMYHVHU52POGAA7VMAU4YM6HTVHC5PA, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.771699443 (376h32m59.542340778s) [noise=true , address=104.36.82.44:4898 , piece_id=X47T6NYNODFSVKHJGWIH2WJT7BA3UVP7BEBYVKPPQBMN4QOQCJ6Q, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.771818129 (376h32m59.542459465s) [noise=true , address=104.36.82.34:4551 , piece_id=HWYKZWYEKMJT3JHDPHRWIAFKP5NZQ72OGR4SPR2I2SRRABLA663A, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.772055795 (376h32m59.54269713s) [noise=true , address=77.90.31.10:4543 , piece_id=QCKPGFCSQ3N57BWXMOSW5LAVA4KOI3EVUFU2OMPN7CMAOYAH4RWQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.772089298 (376h32m59.542730623s) [noise=true , address=104.36.82.23:4459 , piece_id=SOG4BGCMVK36OVMKHGIYGSVGX2ZZ2PWZV74S2LQS6IZQRGDYJVYA, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.772274673 (376h32m59.542916008s) [noise=true , address=77.90.31.15:4715 , piece_id=ITPPGCX32ZLU72XXE7RG6AYE3PM7IILVNPXM5A4NJ2S7P6U4Y2FA, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:25.540057386 (376h33m0.310698721s) [noise=true , address=64.49.13.169:4706 , piece_id=2CKVJCKZJH4F3VC3X7XJOWMYHVHU52POGAA7VMAU4YM6HTVHC5PA, seg_pos={0 0}, upload=301248]: Done uploading piece. err: ecclient: upload failed (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7, address:64.49.13.169:4706): protocol: storage node overloaded, request limit: 2000
Jul 9 01:43:25.878055136 (376h33m0.648696481s) [upload=301248]: Next returned nil segment
Jul 9 01:43:25.878073130 (376h33m0.648714465s) [upload=301248]: Waiting for error group managing segments...
Jul 9 01:43:26.175673001 (376h33m0.946314336s) [noise=true , address=79.127.205.234:4614 , piece_id=JPPMN7KOKXFYF2YDK4RC7AU6KYV2UHQELFEG6ZZ6W5HQ2MKFVN5A, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.774276626 (376h33m1.544917961s) [noise=true , address=77.90.31.12:4622 , piece_id=25K7AOPYFLZ6EDAA3SPHHE6S635PN5VB3ZK4LIULBHKQGYYQ3C7A, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.778019557 (376h33m1.548660892s) [noise=true , address=77.90.31.21:4886 , piece_id=YRDZYEV6KU5YISF4DIS3GTZVBFRCZEOIW2DG7VCKHO26NRXMHMVQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.779099603 (376h33m1.549740938s) [noise=true , address=77.90.31.14:4691 , piece_id=ELMXSLQT6TNOMGNUGM4BEHZ2P7OZWNSX6EOCHDAYOD7OIANFUVHA, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.779161091 (376h33m1.549802426s) [noise=true , address=77.90.31.18:4825 , piece_id=NFOYMUTMKMKTPCAEPBTWL43ISOETOAM4WKVAUNHCETNW75IQLLMQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.780045593 (376h33m1.550686928s) [noise=true , address=77.90.31.14:4677 , piece_id=Z7SGXIO722E3U7PWA4AWOUD5T27VSQUCG3ZBVP22WJRFSRKOEXAQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.795740620 (376h33m1.566381976s) [noise=true , address=77.90.31.10:4543 , piece_id=QCKPGFCSQ3N57BWXMOSW5LAVA4KOI3EVUFU2OMPN7CMAOYAH4RWQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.795758073 (376h33m1.566399408s) [noise=true , address=77.90.31.15:4715 , piece_id=ITPPGCX32ZLU72XXE7RG6AYE3PM7IILVNPXM5A4NJ2S7P6U4Y2FA, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.812585147 (376h33m1.583226483s) [noise=true , address=77.90.31.19:4848 , piece_id=RV7TBRZW2YH64X5CLXFHFTJD2WQCEM3P7OW3NYZ2IWC2E3DJDG5A, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:27.386630478 (376h33m2.157271803s) [noise=true , address=45.91.116.11:4123 , piece_id=WSMWN6I2QVBTO6OCRRDCMT3G3TESW3H2ZET5YNWFUPVENPN3UWKQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:37.603914421 (376h33m12.374555756s) [noise=true , address=104.36.82.34:4551 , piece_id=HWYKZWYEKMJT3JHDPHRWIAFKP5NZQ72OGR4SPR2I2SRRABLA663A, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:41.629246242 (376h33m16.399887567s) [noise=true , address=104.36.82.12:4160 , piece_id=2Z3FQL5GTT26HC2GPKO5YPOLPB2Z75K4UG3FDPMOEDCCD6Q6MULQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:44:10.222815170 (376h33m44.993456505s) [noise=true , address=104.36.82.22:4511 , piece_id=3AIKVWFRA4CLMPRDYFNX6EA7XO22MB7RQEOVNXPUVUCNFAHPSFCQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:44:21.573027994 (376h33m56.343669329s) [noise=true , address=104.36.82.23:4459 , piece_id=SOG4BGCMVK36OVMKHGIYGSVGX2ZZ2PWZV74S2LQS6IZQRGDYJVYA, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:44:21.573054295 (376h33m56.34369563s) [seg_pos={0 0}, upload=301248]: Segment reached optimal threshold of 14 pieces.
Jul 9 01:44:21.573140269 (376h33m56.343781604s) [noise=true , address=104.36.82.40:4955 , piece_id=3TVDS5QR5267W6VKWNXOPW6ZSTXISGXFBIPKK7E3S6ML7C4X4MUQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ssLq5bo7og4z8ZPd28f2H6WUExsWohBBZUnciGaH5KKpcQ25d): protocol: expected piece hash; context canceled; EOF
Jul 9 01:44:21.573153795 (376h33m56.34379513s) [noise=true , address=104.36.82.19:4411 , piece_id=R27NIJUXHNTQWK4IXXSVS2QC535JAZJ3K2WV4VANVOGKRYV7YADA, seg_pos={0 0}, upload=301248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nradx4SMYLkuJw3Ad6o9Sq1iNtQR6GDu8qg7tCzZzNcTjVfyd): protocol: expected piece hash; context canceled; EOF
Jul 9 01:44:21.573259487 (376h33m56.343900822s) [noise=true , address=104.36.82.44:4898 , piece_id=X47T6NYNODFSVKHJGWIH2WJT7BA3UVP7BEBYVKPPQBMN4QOQCJ6Q, seg_pos={0 0}, upload=301248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc): protocol: write tcp 188.166.225.170:40614->104.36.82.44:4898: use of closed network connection; write tcp 188.166.225.170:40614->104.36.82.44:4898: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40614->104.36.82.44:4898: use of closed network connection
Jul 9 01:44:21.573276430 (376h33m56.343917755s) [seg_pos={0 0}, upload=301248]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 01:44:21.573349429 (376h33m56.343990764s) [upload=301248]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 01:44:21.573353537 (376h33m56.343994873s) [upload=301248]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 01:44:21.865577192 (376h33m56.636218527s) [upload=301248]: Done uploading segments...
Jul 9 01:44:21.865594906 (376h33m56.636236232s) [upload=301248]: Upload finished. err: <nil>
Duration: 28.133228863s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [......................X......................X..............] duration:
5.80862239s
FAIL (2025-07-09 01:53:25 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Jul 9 01:41:35.784016383 (376h31m10.554657718s) [upload=301225]: Starting upload
Jul 9 01:41:35.784073732 (376h31m10.554715067s) [upload=301225]: Done starting upload
Jul 9 01:41:35.784180246 (376h31m10.554821571s) [upload=301225]: Uploading segments...
Jul 9 01:41:35.784275688 (376h31m10.554917023s) [upload=301225]: Got next segment. Inline: false
Jul 9 01:41:35.784309523 (376h31m10.554950848s) [upload=301225]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 01:41:35.784314523 (376h31m10.554955858s) [upload=301225]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 01:41:36.035706020 (376h31m10.806347375s) [seg_pos={0 0}, upload=301225]: Begin upload segment...
Jul 9 01:41:36.035755273 (376h31m10.806396598s) [seg_pos={0 0}, upload=301225]: Done begin upload segment.
Jul 9 01:41:36.035768209 (376h31m10.806409574s) [noise=true , address=45.91.116.14:4190 , piece_id=CZGKNHBIJP7HZIC65I6ESRFRQLBKTEAQJVVUQHHZHFO7TNNROJ4A, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.035782255 (376h31m10.80642359s) [noise=true , address=104.36.82.16:4294 , piece_id=CE4MNKGLLBYQAQ7KGVTTGUMWRTD2NTWF55NOPILQC5YPXDQT3Z6A, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.035859012 (376h31m10.806500367s) [noise=true , address=104.36.82.5:4840 , piece_id=QKGEMZVR5P3B6ARIGS4VQDSWGAOVC5DU65Y3VLXUY2MAYVKDUQCQ, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.036404877 (376h31m10.807046232s) [noise=true , address=45.91.116.8:4045 , piece_id=Z37SMFGUPVJVP6RNGUIDH6Q456OJFVZ7SNFFYT3NT3AOXVYR5TXA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.036420547 (376h31m10.807061892s) [noise=true , address=77.90.31.19:4843 , piece_id=JH6TQLWG4LCPENFHL6ORQQBR5YDUOU2QAY7OLQ5KYXOK2DW5DRGA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.036942675 (376h31m10.80758403s) [noise=true , address=104.36.82.9:4007 , piece_id=TMZJKWZDZHE4ANML45KWGRPDG5UOTNIYUZFDH3LJYZHMU6LBNTIA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.037006067 (376h31m10.807647412s) [noise=true , address=45.91.116.10:4082 , piece_id=DNZJWYESTZZY2QVM2SRD32AS6GDWSSO7ODNWMHBXJAGRNQ3FMSMQ, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.037346698 (376h31m10.807988043s) [noise=true , address=45.91.116.6:4975 , piece_id=LDM5B2S4NP43AXC3DJXTUM66Z6EM3SBKPP6JHTMLA7RZATZQ4QQA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.037776601 (376h31m10.808417936s) [noise=true , address=104.36.82.11:4114 , piece_id=5ISRL5WJST2SCUOPBB7KX53T3PKO2CH5X7QKCPGSKSDO3KA5BYHQ, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.038177858 (376h31m10.808819193s) [noise=true , address=104.36.82.14:4222 , piece_id=FU4BHM26LXD5XUTT333YO7ETORBXWP6QWCLUUJI2MBMPWLEFMGAA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.038258122 (376h31m10.808899467s) [noise=true , address=77.90.31.12:4592 , piece_id=J7LR65VEHL5D7XBO2HMEMFZVFBPEES5IL7FQTV5IJK4ZHEHGDKOA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.038562023 (376h31m10.809203359s) [noise=true , address=45.91.116.13:4153 , piece_id=UMTA7DVZBXEYRQQXBO33BCQL4W5L5RKJAYACZHYJMHCX6XORWSWQ, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.038582503 (376h31m10.809223838s) [noise=true , address=104.36.82.2:4730 , piece_id=EVW56CTEQD7BPUSYS2U5NXO4VRG6YZVEITV6OU5XKT3AU3MRWRYQ, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.038924036 (376h31m10.809565371s) [noise=true , address=104.36.82.8:4969 , piece_id=6IKLPKOPP2HBAOC5FAI5IE5SNW2B2UWNWT3Q6S4WWKGDLRCARVWA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.039254127 (376h31m10.809895453s) [noise=true , address=77.90.31.21:4878 , piece_id=4YNDQENFOKMHFJFVWWBC7UK4TQP745OUW7S2P43BICF2SNFAWIHA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.039603647 (376h31m10.810244982s) [noise=true , address=45.91.116.12:4227 , piece_id=23YQLIAPNPNMSA3LL67PNYCRDMWJ4GXZY3VF354XXUH5YEXFWVUA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:37.680432693 (376h31m12.451074028s) [upload=301225]: Next returned nil segment
Jul 9 01:41:37.680461809 (376h31m12.451103144s) [upload=301225]: Waiting for error group managing segments...
Jul 9 01:41:38.011139865 (376h31m12.7817812s) [noise=true , address=104.36.82.16:4294 , piece_id=CE4MNKGLLBYQAQ7KGVTTGUMWRTD2NTWF55NOPILQC5YPXDQT3Z6A, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.014224286 (376h31m12.784865621s) [noise=true , address=104.36.82.9:4007 , piece_id=TMZJKWZDZHE4ANML45KWGRPDG5UOTNIYUZFDH3LJYZHMU6LBNTIA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.116297430 (376h31m12.886938765s) [noise=true , address=45.91.116.6:4975 , piece_id=LDM5B2S4NP43AXC3DJXTUM66Z6EM3SBKPP6JHTMLA7RZATZQ4QQA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.118333195 (376h31m12.88897453s) [noise=true , address=45.91.116.13:4153 , piece_id=UMTA7DVZBXEYRQQXBO33BCQL4W5L5RKJAYACZHYJMHCX6XORWSWQ, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.130186111 (376h31m12.900827446s) [noise=true , address=45.91.116.8:4045 , piece_id=Z37SMFGUPVJVP6RNGUIDH6Q456OJFVZ7SNFFYT3NT3AOXVYR5TXA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.130610163 (376h31m12.901251498s) [noise=true , address=45.91.116.10:4082 , piece_id=DNZJWYESTZZY2QVM2SRD32AS6GDWSSO7ODNWMHBXJAGRNQ3FMSMQ, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.171163518 (376h31m12.941804854s) [noise=true , address=77.90.31.19:4843 , piece_id=JH6TQLWG4LCPENFHL6ORQQBR5YDUOU2QAY7OLQ5KYXOK2DW5DRGA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.187728911 (376h31m12.958370246s) [noise=true , address=77.90.31.12:4592 , piece_id=J7LR65VEHL5D7XBO2HMEMFZVFBPEES5IL7FQTV5IJK4ZHEHGDKOA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.190933191 (376h31m12.961574526s) [noise=true , address=77.90.31.21:4878 , piece_id=4YNDQENFOKMHFJFVWWBC7UK4TQP745OUW7S2P43BICF2SNFAWIHA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.218675778 (376h31m12.989317113s) [noise=true , address=45.91.116.12:4227 , piece_id=23YQLIAPNPNMSA3LL67PNYCRDMWJ4GXZY3VF354XXUH5YEXFWVUA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.225536314 (376h31m12.996177649s) [noise=true , address=45.91.116.14:4190 , piece_id=CZGKNHBIJP7HZIC65I6ESRFRQLBKTEAQJVVUQHHZHFO7TNNROJ4A, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:42:05.141496679 (376h31m39.912138074s) [noise=true , address=104.36.82.8:4969 , piece_id=6IKLPKOPP2HBAOC5FAI5IE5SNW2B2UWNWT3Q6S4WWKGDLRCARVWA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 01:42:05.141543177 (376h31m39.912184512s) [noise=true , address=104.36.82.14:4222 , piece_id=FU4BHM26LXD5XUTT333YO7ETORBXWP6QWCLUUJI2MBMPWLEFMGAA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 188.166.225.170:60278->104.36.82.14:4222: use of closed network connection; write tcp 188.166.225.170:60278->104.36.82.14:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60278->104.36.82.14:4222: use of closed network connection
Jul 9 01:42:05.141602951 (376h31m39.912244286s) [noise=true , address=104.36.82.5:4840 , piece_id=QKGEMZVR5P3B6ARIGS4VQDSWGAOVC5DU65Y3VLXUY2MAYVKDUQCQ, seg_pos={0 0}, upload=301225]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: write tcp 188.166.225.170:35250->104.36.82.5:4840: use of closed network connection; write tcp 188.166.225.170:35250->104.36.82.5:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35250->104.36.82.5:4840: use of closed network connection
Jul 9 01:42:05.141829285 (376h31m39.91247062s) [noise=true , address=104.36.82.11:4114 , piece_id=5ISRL5WJST2SCUOPBB7KX53T3PKO2CH5X7QKCPGSKSDO3KA5BYHQ, seg_pos={0 0}, upload=301225]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 188.166.225.170:54484->104.36.82.11:4114: use of closed network connection; write tcp 188.166.225.170:54484->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54484->104.36.82.11:4114: use of closed network connection
Jul 9 01:42:05.141852700 (376h31m39.912494025s) [noise=true , address=104.36.82.2:4730 , piece_id=EVW56CTEQD7BPUSYS2U5NXO4VRG6YZVEITV6OU5XKT3AU3MRWRYQ, seg_pos={0 0}, upload=301225]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 188.166.225.170:59748->104.36.82.2:4730: use of closed network connection; write tcp 188.166.225.170:59748->104.36.82.2:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59748->104.36.82.2:4730: use of closed network connection
Jul 9 01:42:05.141879792 (376h31m39.912521127s) [seg_pos={0 0}, upload=301225]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 9 01:42:05.366924712 (376h31m40.137566037s) [upload=301225]: Done uploading segments...
Jul 9 01:42:05.366949890 (376h31m40.137591225s) [upload=301225]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 5.80865301s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [..X...X.X....X...X.X..X...X..X............X......X..X...X.X.] duration:
5.903209485s
FAIL (2025-07-09 01:53:11 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.903234333s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
14 out of 60 is failed (threshold=3) [..................X......XX.XX..X......XXX....XX.....XX...X.] duration:
56.848938312s
FAIL (2025-07-09 01:53:09 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 56.848951038s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [.....X..........................X......X..........X.........] duration:
42.489603515s
FAIL (2025-07-09 01:52:50 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
s) [upload=301248]: Got next segment. Inline: false
Jul 9 01:43:24.322645491 (376h32m59.093286826s) [upload=301248]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 01:43:24.322650441 (376h32m59.093291777s) [upload=301248]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 01:43:24.769774209 (376h32m59.540415544s) [seg_pos={0 0}, upload=301248]: Begin upload segment...
Jul 9 01:43:24.769830437 (376h32m59.540471772s) [seg_pos={0 0}, upload=301248]: Done begin upload segment.
Jul 9 01:43:24.769852048 (376h32m59.540493373s) [noise=true , address=79.127.205.234:4614 , piece_id=JPPMN7KOKXFYF2YDK4RC7AU6KYV2UHQELFEG6ZZ6W5HQ2MKFVN5A, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.769864432 (376h32m59.540505767s) [noise=true , address=45.91.116.11:4123 , piece_id=WSMWN6I2QVBTO6OCRRDCMT3G3TESW3H2ZET5YNWFUPVENPN3UWKQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.770250630 (376h32m59.540891965s) [noise=true , address=104.36.82.22:4511 , piece_id=3AIKVWFRA4CLMPRDYFNX6EA7XO22MB7RQEOVNXPUVUCNFAHPSFCQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.770361543 (376h32m59.541002878s) [noise=true , address=77.90.31.21:4886 , piece_id=YRDZYEV6KU5YISF4DIS3GTZVBFRCZEOIW2DG7VCKHO26NRXMHMVQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.770542389 (376h32m59.541183724s) [noise=true , address=77.90.31.19:4848 , piece_id=RV7TBRZW2YH64X5CLXFHFTJD2WQCEM3P7OW3NYZ2IWC2E3DJDG5A, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.770626239 (376h32m59.541267594s) [noise=true , address=104.36.82.19:4411 , piece_id=R27NIJUXHNTQWK4IXXSVS2QC535JAZJ3K2WV4VANVOGKRYV7YADA, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.770818237 (376h32m59.541459572s) [noise=true , address=104.36.82.40:4955 , piece_id=3TVDS5QR5267W6VKWNXOPW6ZSTXISGXFBIPKK7E3S6ML7C4X4MUQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.770918087 (376h32m59.541559422s) [noise=true , address=104.36.82.12:4160 , piece_id=2Z3FQL5GTT26HC2GPKO5YPOLPB2Z75K4UG3FDPMOEDCCD6Q6MULQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.771127618 (376h32m59.541768953s) [noise=true , address=77.90.31.14:4677 , piece_id=Z7SGXIO722E3U7PWA4AWOUD5T27VSQUCG3ZBVP22WJRFSRKOEXAQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.771146836 (376h32m59.541788171s) [noise=true , address=77.90.31.12:4622 , piece_id=25K7AOPYFLZ6EDAA3SPHHE6S635PN5VB3ZK4LIULBHKQGYYQ3C7A, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.771367858 (376h32m59.542009193s) [noise=true , address=77.90.31.18:4825 , piece_id=NFOYMUTMKMKTPCAEPBTWL43ISOETOAM4WKVAUNHCETNW75IQLLMQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.771421060 (376h32m59.542062395s) [noise=true , address=77.90.31.14:4691 , piece_id=ELMXSLQT6TNOMGNUGM4BEHZ2P7OZWNSX6EOCHDAYOD7OIANFUVHA, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.771586527 (376h32m59.542227852s) [noise=true , address=64.49.13.169:4706 , piece_id=2CKVJCKZJH4F3VC3X7XJOWMYHVHU52POGAA7VMAU4YM6HTVHC5PA, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.771699443 (376h32m59.542340778s) [noise=true , address=104.36.82.44:4898 , piece_id=X47T6NYNODFSVKHJGWIH2WJT7BA3UVP7BEBYVKPPQBMN4QOQCJ6Q, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.771818129 (376h32m59.542459465s) [noise=true , address=104.36.82.34:4551 , piece_id=HWYKZWYEKMJT3JHDPHRWIAFKP5NZQ72OGR4SPR2I2SRRABLA663A, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.772055795 (376h32m59.54269713s) [noise=true , address=77.90.31.10:4543 , piece_id=QCKPGFCSQ3N57BWXMOSW5LAVA4KOI3EVUFU2OMPN7CMAOYAH4RWQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.772089298 (376h32m59.542730623s) [noise=true , address=104.36.82.23:4459 , piece_id=SOG4BGCMVK36OVMKHGIYGSVGX2ZZ2PWZV74S2LQS6IZQRGDYJVYA, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.772274673 (376h32m59.542916008s) [noise=true , address=77.90.31.15:4715 , piece_id=ITPPGCX32ZLU72XXE7RG6AYE3PM7IILVNPXM5A4NJ2S7P6U4Y2FA, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:25.540057386 (376h33m0.310698721s) [noise=true , address=64.49.13.169:4706 , piece_id=2CKVJCKZJH4F3VC3X7XJOWMYHVHU52POGAA7VMAU4YM6HTVHC5PA, seg_pos={0 0}, upload=301248]: Done uploading piece. err: ecclient: upload failed (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7, address:64.49.13.169:4706): protocol: storage node overloaded, request limit: 2000
Jul 9 01:43:25.878055136 (376h33m0.648696481s) [upload=301248]: Next returned nil segment
Jul 9 01:43:25.878073130 (376h33m0.648714465s) [upload=301248]: Waiting for error group managing segments...
Jul 9 01:43:26.175673001 (376h33m0.946314336s) [noise=true , address=79.127.205.234:4614 , piece_id=JPPMN7KOKXFYF2YDK4RC7AU6KYV2UHQELFEG6ZZ6W5HQ2MKFVN5A, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.774276626 (376h33m1.544917961s) [noise=true , address=77.90.31.12:4622 , piece_id=25K7AOPYFLZ6EDAA3SPHHE6S635PN5VB3ZK4LIULBHKQGYYQ3C7A, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.778019557 (376h33m1.548660892s) [noise=true , address=77.90.31.21:4886 , piece_id=YRDZYEV6KU5YISF4DIS3GTZVBFRCZEOIW2DG7VCKHO26NRXMHMVQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.779099603 (376h33m1.549740938s) [noise=true , address=77.90.31.14:4691 , piece_id=ELMXSLQT6TNOMGNUGM4BEHZ2P7OZWNSX6EOCHDAYOD7OIANFUVHA, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.779161091 (376h33m1.549802426s) [noise=true , address=77.90.31.18:4825 , piece_id=NFOYMUTMKMKTPCAEPBTWL43ISOETOAM4WKVAUNHCETNW75IQLLMQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.780045593 (376h33m1.550686928s) [noise=true , address=77.90.31.14:4677 , piece_id=Z7SGXIO722E3U7PWA4AWOUD5T27VSQUCG3ZBVP22WJRFSRKOEXAQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.795740620 (376h33m1.566381976s) [noise=true , address=77.90.31.10:4543 , piece_id=QCKPGFCSQ3N57BWXMOSW5LAVA4KOI3EVUFU2OMPN7CMAOYAH4RWQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.795758073 (376h33m1.566399408s) [noise=true , address=77.90.31.15:4715 , piece_id=ITPPGCX32ZLU72XXE7RG6AYE3PM7IILVNPXM5A4NJ2S7P6U4Y2FA, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.812585147 (376h33m1.583226483s) [noise=true , address=77.90.31.19:4848 , piece_id=RV7TBRZW2YH64X5CLXFHFTJD2WQCEM3P7OW3NYZ2IWC2E3DJDG5A, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:27.386630478 (376h33m2.157271803s) [noise=true , address=45.91.116.11:4123 , piece_id=WSMWN6I2QVBTO6OCRRDCMT3G3TESW3H2ZET5YNWFUPVENPN3UWKQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:37.603914421 (376h33m12.374555756s) [noise=true , address=104.36.82.34:4551 , piece_id=HWYKZWYEKMJT3JHDPHRWIAFKP5NZQ72OGR4SPR2I2SRRABLA663A, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:41.629246242 (376h33m16.399887567s) [noise=true , address=104.36.82.12:4160 , piece_id=2Z3FQL5GTT26HC2GPKO5YPOLPB2Z75K4UG3FDPMOEDCCD6Q6MULQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:44:10.222815170 (376h33m44.993456505s) [noise=true , address=104.36.82.22:4511 , piece_id=3AIKVWFRA4CLMPRDYFNX6EA7XO22MB7RQEOVNXPUVUCNFAHPSFCQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:44:21.573027994 (376h33m56.343669329s) [noise=true , address=104.36.82.23:4459 , piece_id=SOG4BGCMVK36OVMKHGIYGSVGX2ZZ2PWZV74S2LQS6IZQRGDYJVYA, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:44:21.573054295 (376h33m56.34369563s) [seg_pos={0 0}, upload=301248]: Segment reached optimal threshold of 14 pieces.
Jul 9 01:44:21.573140269 (376h33m56.343781604s) [noise=true , address=104.36.82.40:4955 , piece_id=3TVDS5QR5267W6VKWNXOPW6ZSTXISGXFBIPKK7E3S6ML7C4X4MUQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ssLq5bo7og4z8ZPd28f2H6WUExsWohBBZUnciGaH5KKpcQ25d): protocol: expected piece hash; context canceled; EOF
Jul 9 01:44:21.573153795 (376h33m56.34379513s) [noise=true , address=104.36.82.19:4411 , piece_id=R27NIJUXHNTQWK4IXXSVS2QC535JAZJ3K2WV4VANVOGKRYV7YADA, seg_pos={0 0}, upload=301248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nradx4SMYLkuJw3Ad6o9Sq1iNtQR6GDu8qg7tCzZzNcTjVfyd): protocol: expected piece hash; context canceled; EOF
Jul 9 01:44:21.573259487 (376h33m56.343900822s) [noise=true , address=104.36.82.44:4898 , piece_id=X47T6NYNODFSVKHJGWIH2WJT7BA3UVP7BEBYVKPPQBMN4QOQCJ6Q, seg_pos={0 0}, upload=301248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc): protocol: write tcp 188.166.225.170:40614->104.36.82.44:4898: use of closed network connection; write tcp 188.166.225.170:40614->104.36.82.44:4898: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40614->104.36.82.44:4898: use of closed network connection
Jul 9 01:44:21.573276430 (376h33m56.343917755s) [seg_pos={0 0}, upload=301248]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 01:44:21.573349429 (376h33m56.343990764s) [upload=301248]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 01:44:21.573353537 (376h33m56.343994873s) [upload=301248]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 01:44:21.865577192 (376h33m56.636218527s) [upload=301248]: Done uploading segments...
Jul 9 01:44:21.865594906 (376h33m56.636236232s) [upload=301248]: Upload finished. err: <nil>
Duration: 42.489629974s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [...X...X.X....X...X.X..X...X..X............X......X..X...X.X] duration:
30.000614646s
FAIL (2025-07-09 01:52:48 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000638812s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
14 out of 60 is failed (threshold=3) [...................X......XX.XX..X......XXX....XX.....XX...X] duration:
1m0.000399999s
FAIL (2025-07-09 01:52:12 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.000417774s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.......................X......................X.............] duration:
5.671534318s
FAIL (2025-07-09 01:52:11 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Jul 9 01:41:35.784016383 (376h31m10.554657718s) [upload=301225]: Starting upload
Jul 9 01:41:35.784073732 (376h31m10.554715067s) [upload=301225]: Done starting upload
Jul 9 01:41:35.784180246 (376h31m10.554821571s) [upload=301225]: Uploading segments...
Jul 9 01:41:35.784275688 (376h31m10.554917023s) [upload=301225]: Got next segment. Inline: false
Jul 9 01:41:35.784309523 (376h31m10.554950848s) [upload=301225]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 01:41:35.784314523 (376h31m10.554955858s) [upload=301225]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 01:41:36.035706020 (376h31m10.806347375s) [seg_pos={0 0}, upload=301225]: Begin upload segment...
Jul 9 01:41:36.035755273 (376h31m10.806396598s) [seg_pos={0 0}, upload=301225]: Done begin upload segment.
Jul 9 01:41:36.035768209 (376h31m10.806409574s) [noise=true , address=45.91.116.14:4190 , piece_id=CZGKNHBIJP7HZIC65I6ESRFRQLBKTEAQJVVUQHHZHFO7TNNROJ4A, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.035782255 (376h31m10.80642359s) [noise=true , address=104.36.82.16:4294 , piece_id=CE4MNKGLLBYQAQ7KGVTTGUMWRTD2NTWF55NOPILQC5YPXDQT3Z6A, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.035859012 (376h31m10.806500367s) [noise=true , address=104.36.82.5:4840 , piece_id=QKGEMZVR5P3B6ARIGS4VQDSWGAOVC5DU65Y3VLXUY2MAYVKDUQCQ, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.036404877 (376h31m10.807046232s) [noise=true , address=45.91.116.8:4045 , piece_id=Z37SMFGUPVJVP6RNGUIDH6Q456OJFVZ7SNFFYT3NT3AOXVYR5TXA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.036420547 (376h31m10.807061892s) [noise=true , address=77.90.31.19:4843 , piece_id=JH6TQLWG4LCPENFHL6ORQQBR5YDUOU2QAY7OLQ5KYXOK2DW5DRGA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.036942675 (376h31m10.80758403s) [noise=true , address=104.36.82.9:4007 , piece_id=TMZJKWZDZHE4ANML45KWGRPDG5UOTNIYUZFDH3LJYZHMU6LBNTIA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.037006067 (376h31m10.807647412s) [noise=true , address=45.91.116.10:4082 , piece_id=DNZJWYESTZZY2QVM2SRD32AS6GDWSSO7ODNWMHBXJAGRNQ3FMSMQ, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.037346698 (376h31m10.807988043s) [noise=true , address=45.91.116.6:4975 , piece_id=LDM5B2S4NP43AXC3DJXTUM66Z6EM3SBKPP6JHTMLA7RZATZQ4QQA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.037776601 (376h31m10.808417936s) [noise=true , address=104.36.82.11:4114 , piece_id=5ISRL5WJST2SCUOPBB7KX53T3PKO2CH5X7QKCPGSKSDO3KA5BYHQ, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.038177858 (376h31m10.808819193s) [noise=true , address=104.36.82.14:4222 , piece_id=FU4BHM26LXD5XUTT333YO7ETORBXWP6QWCLUUJI2MBMPWLEFMGAA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.038258122 (376h31m10.808899467s) [noise=true , address=77.90.31.12:4592 , piece_id=J7LR65VEHL5D7XBO2HMEMFZVFBPEES5IL7FQTV5IJK4ZHEHGDKOA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.038562023 (376h31m10.809203359s) [noise=true , address=45.91.116.13:4153 , piece_id=UMTA7DVZBXEYRQQXBO33BCQL4W5L5RKJAYACZHYJMHCX6XORWSWQ, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.038582503 (376h31m10.809223838s) [noise=true , address=104.36.82.2:4730 , piece_id=EVW56CTEQD7BPUSYS2U5NXO4VRG6YZVEITV6OU5XKT3AU3MRWRYQ, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.038924036 (376h31m10.809565371s) [noise=true , address=104.36.82.8:4969 , piece_id=6IKLPKOPP2HBAOC5FAI5IE5SNW2B2UWNWT3Q6S4WWKGDLRCARVWA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.039254127 (376h31m10.809895453s) [noise=true , address=77.90.31.21:4878 , piece_id=4YNDQENFOKMHFJFVWWBC7UK4TQP745OUW7S2P43BICF2SNFAWIHA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.039603647 (376h31m10.810244982s) [noise=true , address=45.91.116.12:4227 , piece_id=23YQLIAPNPNMSA3LL67PNYCRDMWJ4GXZY3VF354XXUH5YEXFWVUA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:37.680432693 (376h31m12.451074028s) [upload=301225]: Next returned nil segment
Jul 9 01:41:37.680461809 (376h31m12.451103144s) [upload=301225]: Waiting for error group managing segments...
Jul 9 01:41:38.011139865 (376h31m12.7817812s) [noise=true , address=104.36.82.16:4294 , piece_id=CE4MNKGLLBYQAQ7KGVTTGUMWRTD2NTWF55NOPILQC5YPXDQT3Z6A, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.014224286 (376h31m12.784865621s) [noise=true , address=104.36.82.9:4007 , piece_id=TMZJKWZDZHE4ANML45KWGRPDG5UOTNIYUZFDH3LJYZHMU6LBNTIA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.116297430 (376h31m12.886938765s) [noise=true , address=45.91.116.6:4975 , piece_id=LDM5B2S4NP43AXC3DJXTUM66Z6EM3SBKPP6JHTMLA7RZATZQ4QQA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.118333195 (376h31m12.88897453s) [noise=true , address=45.91.116.13:4153 , piece_id=UMTA7DVZBXEYRQQXBO33BCQL4W5L5RKJAYACZHYJMHCX6XORWSWQ, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.130186111 (376h31m12.900827446s) [noise=true , address=45.91.116.8:4045 , piece_id=Z37SMFGUPVJVP6RNGUIDH6Q456OJFVZ7SNFFYT3NT3AOXVYR5TXA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.130610163 (376h31m12.901251498s) [noise=true , address=45.91.116.10:4082 , piece_id=DNZJWYESTZZY2QVM2SRD32AS6GDWSSO7ODNWMHBXJAGRNQ3FMSMQ, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.171163518 (376h31m12.941804854s) [noise=true , address=77.90.31.19:4843 , piece_id=JH6TQLWG4LCPENFHL6ORQQBR5YDUOU2QAY7OLQ5KYXOK2DW5DRGA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.187728911 (376h31m12.958370246s) [noise=true , address=77.90.31.12:4592 , piece_id=J7LR65VEHL5D7XBO2HMEMFZVFBPEES5IL7FQTV5IJK4ZHEHGDKOA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.190933191 (376h31m12.961574526s) [noise=true , address=77.90.31.21:4878 , piece_id=4YNDQENFOKMHFJFVWWBC7UK4TQP745OUW7S2P43BICF2SNFAWIHA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.218675778 (376h31m12.989317113s) [noise=true , address=45.91.116.12:4227 , piece_id=23YQLIAPNPNMSA3LL67PNYCRDMWJ4GXZY3VF354XXUH5YEXFWVUA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.225536314 (376h31m12.996177649s) [noise=true , address=45.91.116.14:4190 , piece_id=CZGKNHBIJP7HZIC65I6ESRFRQLBKTEAQJVVUQHHZHFO7TNNROJ4A, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:42:05.141496679 (376h31m39.912138074s) [noise=true , address=104.36.82.8:4969 , piece_id=6IKLPKOPP2HBAOC5FAI5IE5SNW2B2UWNWT3Q6S4WWKGDLRCARVWA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 01:42:05.141543177 (376h31m39.912184512s) [noise=true , address=104.36.82.14:4222 , piece_id=FU4BHM26LXD5XUTT333YO7ETORBXWP6QWCLUUJI2MBMPWLEFMGAA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 188.166.225.170:60278->104.36.82.14:4222: use of closed network connection; write tcp 188.166.225.170:60278->104.36.82.14:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60278->104.36.82.14:4222: use of closed network connection
Jul 9 01:42:05.141602951 (376h31m39.912244286s) [noise=true , address=104.36.82.5:4840 , piece_id=QKGEMZVR5P3B6ARIGS4VQDSWGAOVC5DU65Y3VLXUY2MAYVKDUQCQ, seg_pos={0 0}, upload=301225]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: write tcp 188.166.225.170:35250->104.36.82.5:4840: use of closed network connection; write tcp 188.166.225.170:35250->104.36.82.5:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35250->104.36.82.5:4840: use of closed network connection
Jul 9 01:42:05.141829285 (376h31m39.91247062s) [noise=true , address=104.36.82.11:4114 , piece_id=5ISRL5WJST2SCUOPBB7KX53T3PKO2CH5X7QKCPGSKSDO3KA5BYHQ, seg_pos={0 0}, upload=301225]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 188.166.225.170:54484->104.36.82.11:4114: use of closed network connection; write tcp 188.166.225.170:54484->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54484->104.36.82.11:4114: use of closed network connection
Jul 9 01:42:05.141852700 (376h31m39.912494025s) [noise=true , address=104.36.82.2:4730 , piece_id=EVW56CTEQD7BPUSYS2U5NXO4VRG6YZVEITV6OU5XKT3AU3MRWRYQ, seg_pos={0 0}, upload=301225]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 188.166.225.170:59748->104.36.82.2:4730: use of closed network connection; write tcp 188.166.225.170:59748->104.36.82.2:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59748->104.36.82.2:4730: use of closed network connection
Jul 9 01:42:05.141879792 (376h31m39.912521127s) [seg_pos={0 0}, upload=301225]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 9 01:42:05.366924712 (376h31m40.137566037s) [upload=301225]: Done uploading segments...
Jul 9 01:42:05.366949890 (376h31m40.137591225s) [upload=301225]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 5.671559545s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [....X...X.X....X...X.X..X...X..X............X......X..X...X.] duration:
10.930353562s
FAIL (2025-07-09 01:51:29 UTC)
orbiter warning: context deadline exceeded
Duration: 10.930392335s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [......X..........................X......X..........X........] duration:
9.899763691s
FAIL (2025-07-09 01:51:25 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
s) [upload=301248]: Got next segment. Inline: false
Jul 9 01:43:24.322645491 (376h32m59.093286826s) [upload=301248]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 01:43:24.322650441 (376h32m59.093291777s) [upload=301248]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 01:43:24.769774209 (376h32m59.540415544s) [seg_pos={0 0}, upload=301248]: Begin upload segment...
Jul 9 01:43:24.769830437 (376h32m59.540471772s) [seg_pos={0 0}, upload=301248]: Done begin upload segment.
Jul 9 01:43:24.769852048 (376h32m59.540493373s) [noise=true , address=79.127.205.234:4614 , piece_id=JPPMN7KOKXFYF2YDK4RC7AU6KYV2UHQELFEG6ZZ6W5HQ2MKFVN5A, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.769864432 (376h32m59.540505767s) [noise=true , address=45.91.116.11:4123 , piece_id=WSMWN6I2QVBTO6OCRRDCMT3G3TESW3H2ZET5YNWFUPVENPN3UWKQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.770250630 (376h32m59.540891965s) [noise=true , address=104.36.82.22:4511 , piece_id=3AIKVWFRA4CLMPRDYFNX6EA7XO22MB7RQEOVNXPUVUCNFAHPSFCQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.770361543 (376h32m59.541002878s) [noise=true , address=77.90.31.21:4886 , piece_id=YRDZYEV6KU5YISF4DIS3GTZVBFRCZEOIW2DG7VCKHO26NRXMHMVQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.770542389 (376h32m59.541183724s) [noise=true , address=77.90.31.19:4848 , piece_id=RV7TBRZW2YH64X5CLXFHFTJD2WQCEM3P7OW3NYZ2IWC2E3DJDG5A, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.770626239 (376h32m59.541267594s) [noise=true , address=104.36.82.19:4411 , piece_id=R27NIJUXHNTQWK4IXXSVS2QC535JAZJ3K2WV4VANVOGKRYV7YADA, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.770818237 (376h32m59.541459572s) [noise=true , address=104.36.82.40:4955 , piece_id=3TVDS5QR5267W6VKWNXOPW6ZSTXISGXFBIPKK7E3S6ML7C4X4MUQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.770918087 (376h32m59.541559422s) [noise=true , address=104.36.82.12:4160 , piece_id=2Z3FQL5GTT26HC2GPKO5YPOLPB2Z75K4UG3FDPMOEDCCD6Q6MULQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.771127618 (376h32m59.541768953s) [noise=true , address=77.90.31.14:4677 , piece_id=Z7SGXIO722E3U7PWA4AWOUD5T27VSQUCG3ZBVP22WJRFSRKOEXAQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.771146836 (376h32m59.541788171s) [noise=true , address=77.90.31.12:4622 , piece_id=25K7AOPYFLZ6EDAA3SPHHE6S635PN5VB3ZK4LIULBHKQGYYQ3C7A, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.771367858 (376h32m59.542009193s) [noise=true , address=77.90.31.18:4825 , piece_id=NFOYMUTMKMKTPCAEPBTWL43ISOETOAM4WKVAUNHCETNW75IQLLMQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.771421060 (376h32m59.542062395s) [noise=true , address=77.90.31.14:4691 , piece_id=ELMXSLQT6TNOMGNUGM4BEHZ2P7OZWNSX6EOCHDAYOD7OIANFUVHA, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.771586527 (376h32m59.542227852s) [noise=true , address=64.49.13.169:4706 , piece_id=2CKVJCKZJH4F3VC3X7XJOWMYHVHU52POGAA7VMAU4YM6HTVHC5PA, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.771699443 (376h32m59.542340778s) [noise=true , address=104.36.82.44:4898 , piece_id=X47T6NYNODFSVKHJGWIH2WJT7BA3UVP7BEBYVKPPQBMN4QOQCJ6Q, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.771818129 (376h32m59.542459465s) [noise=true , address=104.36.82.34:4551 , piece_id=HWYKZWYEKMJT3JHDPHRWIAFKP5NZQ72OGR4SPR2I2SRRABLA663A, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.772055795 (376h32m59.54269713s) [noise=true , address=77.90.31.10:4543 , piece_id=QCKPGFCSQ3N57BWXMOSW5LAVA4KOI3EVUFU2OMPN7CMAOYAH4RWQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.772089298 (376h32m59.542730623s) [noise=true , address=104.36.82.23:4459 , piece_id=SOG4BGCMVK36OVMKHGIYGSVGX2ZZ2PWZV74S2LQS6IZQRGDYJVYA, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.772274673 (376h32m59.542916008s) [noise=true , address=77.90.31.15:4715 , piece_id=ITPPGCX32ZLU72XXE7RG6AYE3PM7IILVNPXM5A4NJ2S7P6U4Y2FA, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:25.540057386 (376h33m0.310698721s) [noise=true , address=64.49.13.169:4706 , piece_id=2CKVJCKZJH4F3VC3X7XJOWMYHVHU52POGAA7VMAU4YM6HTVHC5PA, seg_pos={0 0}, upload=301248]: Done uploading piece. err: ecclient: upload failed (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7, address:64.49.13.169:4706): protocol: storage node overloaded, request limit: 2000
Jul 9 01:43:25.878055136 (376h33m0.648696481s) [upload=301248]: Next returned nil segment
Jul 9 01:43:25.878073130 (376h33m0.648714465s) [upload=301248]: Waiting for error group managing segments...
Jul 9 01:43:26.175673001 (376h33m0.946314336s) [noise=true , address=79.127.205.234:4614 , piece_id=JPPMN7KOKXFYF2YDK4RC7AU6KYV2UHQELFEG6ZZ6W5HQ2MKFVN5A, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.774276626 (376h33m1.544917961s) [noise=true , address=77.90.31.12:4622 , piece_id=25K7AOPYFLZ6EDAA3SPHHE6S635PN5VB3ZK4LIULBHKQGYYQ3C7A, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.778019557 (376h33m1.548660892s) [noise=true , address=77.90.31.21:4886 , piece_id=YRDZYEV6KU5YISF4DIS3GTZVBFRCZEOIW2DG7VCKHO26NRXMHMVQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.779099603 (376h33m1.549740938s) [noise=true , address=77.90.31.14:4691 , piece_id=ELMXSLQT6TNOMGNUGM4BEHZ2P7OZWNSX6EOCHDAYOD7OIANFUVHA, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.779161091 (376h33m1.549802426s) [noise=true , address=77.90.31.18:4825 , piece_id=NFOYMUTMKMKTPCAEPBTWL43ISOETOAM4WKVAUNHCETNW75IQLLMQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.780045593 (376h33m1.550686928s) [noise=true , address=77.90.31.14:4677 , piece_id=Z7SGXIO722E3U7PWA4AWOUD5T27VSQUCG3ZBVP22WJRFSRKOEXAQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.795740620 (376h33m1.566381976s) [noise=true , address=77.90.31.10:4543 , piece_id=QCKPGFCSQ3N57BWXMOSW5LAVA4KOI3EVUFU2OMPN7CMAOYAH4RWQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.795758073 (376h33m1.566399408s) [noise=true , address=77.90.31.15:4715 , piece_id=ITPPGCX32ZLU72XXE7RG6AYE3PM7IILVNPXM5A4NJ2S7P6U4Y2FA, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.812585147 (376h33m1.583226483s) [noise=true , address=77.90.31.19:4848 , piece_id=RV7TBRZW2YH64X5CLXFHFTJD2WQCEM3P7OW3NYZ2IWC2E3DJDG5A, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:27.386630478 (376h33m2.157271803s) [noise=true , address=45.91.116.11:4123 , piece_id=WSMWN6I2QVBTO6OCRRDCMT3G3TESW3H2ZET5YNWFUPVENPN3UWKQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:37.603914421 (376h33m12.374555756s) [noise=true , address=104.36.82.34:4551 , piece_id=HWYKZWYEKMJT3JHDPHRWIAFKP5NZQ72OGR4SPR2I2SRRABLA663A, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:41.629246242 (376h33m16.399887567s) [noise=true , address=104.36.82.12:4160 , piece_id=2Z3FQL5GTT26HC2GPKO5YPOLPB2Z75K4UG3FDPMOEDCCD6Q6MULQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:44:10.222815170 (376h33m44.993456505s) [noise=true , address=104.36.82.22:4511 , piece_id=3AIKVWFRA4CLMPRDYFNX6EA7XO22MB7RQEOVNXPUVUCNFAHPSFCQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:44:21.573027994 (376h33m56.343669329s) [noise=true , address=104.36.82.23:4459 , piece_id=SOG4BGCMVK36OVMKHGIYGSVGX2ZZ2PWZV74S2LQS6IZQRGDYJVYA, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:44:21.573054295 (376h33m56.34369563s) [seg_pos={0 0}, upload=301248]: Segment reached optimal threshold of 14 pieces.
Jul 9 01:44:21.573140269 (376h33m56.343781604s) [noise=true , address=104.36.82.40:4955 , piece_id=3TVDS5QR5267W6VKWNXOPW6ZSTXISGXFBIPKK7E3S6ML7C4X4MUQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ssLq5bo7og4z8ZPd28f2H6WUExsWohBBZUnciGaH5KKpcQ25d): protocol: expected piece hash; context canceled; EOF
Jul 9 01:44:21.573153795 (376h33m56.34379513s) [noise=true , address=104.36.82.19:4411 , piece_id=R27NIJUXHNTQWK4IXXSVS2QC535JAZJ3K2WV4VANVOGKRYV7YADA, seg_pos={0 0}, upload=301248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nradx4SMYLkuJw3Ad6o9Sq1iNtQR6GDu8qg7tCzZzNcTjVfyd): protocol: expected piece hash; context canceled; EOF
Jul 9 01:44:21.573259487 (376h33m56.343900822s) [noise=true , address=104.36.82.44:4898 , piece_id=X47T6NYNODFSVKHJGWIH2WJT7BA3UVP7BEBYVKPPQBMN4QOQCJ6Q, seg_pos={0 0}, upload=301248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc): protocol: write tcp 188.166.225.170:40614->104.36.82.44:4898: use of closed network connection; write tcp 188.166.225.170:40614->104.36.82.44:4898: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40614->104.36.82.44:4898: use of closed network connection
Jul 9 01:44:21.573276430 (376h33m56.343917755s) [seg_pos={0 0}, upload=301248]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 01:44:21.573349429 (376h33m56.343990764s) [upload=301248]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 01:44:21.573353537 (376h33m56.343994873s) [upload=301248]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 01:44:21.865577192 (376h33m56.636218527s) [upload=301248]: Done uploading segments...
Jul 9 01:44:21.865594906 (376h33m56.636236232s) [upload=301248]: Upload finished. err: <nil>
Duration: 9.899792958s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
13 out of 60 is failed (threshold=3) [....................X......XX.XX..X......XXX....XX.....XX...] duration:
41.659003704s
FAIL (2025-07-09 01:51:03 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 41.659025094s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [........................X......................X............] duration:
5.758600676s
FAIL (2025-07-09 01:50:59 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Jul 9 01:41:35.784016383 (376h31m10.554657718s) [upload=301225]: Starting upload
Jul 9 01:41:35.784073732 (376h31m10.554715067s) [upload=301225]: Done starting upload
Jul 9 01:41:35.784180246 (376h31m10.554821571s) [upload=301225]: Uploading segments...
Jul 9 01:41:35.784275688 (376h31m10.554917023s) [upload=301225]: Got next segment. Inline: false
Jul 9 01:41:35.784309523 (376h31m10.554950848s) [upload=301225]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 01:41:35.784314523 (376h31m10.554955858s) [upload=301225]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 01:41:36.035706020 (376h31m10.806347375s) [seg_pos={0 0}, upload=301225]: Begin upload segment...
Jul 9 01:41:36.035755273 (376h31m10.806396598s) [seg_pos={0 0}, upload=301225]: Done begin upload segment.
Jul 9 01:41:36.035768209 (376h31m10.806409574s) [noise=true , address=45.91.116.14:4190 , piece_id=CZGKNHBIJP7HZIC65I6ESRFRQLBKTEAQJVVUQHHZHFO7TNNROJ4A, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.035782255 (376h31m10.80642359s) [noise=true , address=104.36.82.16:4294 , piece_id=CE4MNKGLLBYQAQ7KGVTTGUMWRTD2NTWF55NOPILQC5YPXDQT3Z6A, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.035859012 (376h31m10.806500367s) [noise=true , address=104.36.82.5:4840 , piece_id=QKGEMZVR5P3B6ARIGS4VQDSWGAOVC5DU65Y3VLXUY2MAYVKDUQCQ, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.036404877 (376h31m10.807046232s) [noise=true , address=45.91.116.8:4045 , piece_id=Z37SMFGUPVJVP6RNGUIDH6Q456OJFVZ7SNFFYT3NT3AOXVYR5TXA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.036420547 (376h31m10.807061892s) [noise=true , address=77.90.31.19:4843 , piece_id=JH6TQLWG4LCPENFHL6ORQQBR5YDUOU2QAY7OLQ5KYXOK2DW5DRGA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.036942675 (376h31m10.80758403s) [noise=true , address=104.36.82.9:4007 , piece_id=TMZJKWZDZHE4ANML45KWGRPDG5UOTNIYUZFDH3LJYZHMU6LBNTIA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.037006067 (376h31m10.807647412s) [noise=true , address=45.91.116.10:4082 , piece_id=DNZJWYESTZZY2QVM2SRD32AS6GDWSSO7ODNWMHBXJAGRNQ3FMSMQ, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.037346698 (376h31m10.807988043s) [noise=true , address=45.91.116.6:4975 , piece_id=LDM5B2S4NP43AXC3DJXTUM66Z6EM3SBKPP6JHTMLA7RZATZQ4QQA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.037776601 (376h31m10.808417936s) [noise=true , address=104.36.82.11:4114 , piece_id=5ISRL5WJST2SCUOPBB7KX53T3PKO2CH5X7QKCPGSKSDO3KA5BYHQ, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.038177858 (376h31m10.808819193s) [noise=true , address=104.36.82.14:4222 , piece_id=FU4BHM26LXD5XUTT333YO7ETORBXWP6QWCLUUJI2MBMPWLEFMGAA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.038258122 (376h31m10.808899467s) [noise=true , address=77.90.31.12:4592 , piece_id=J7LR65VEHL5D7XBO2HMEMFZVFBPEES5IL7FQTV5IJK4ZHEHGDKOA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.038562023 (376h31m10.809203359s) [noise=true , address=45.91.116.13:4153 , piece_id=UMTA7DVZBXEYRQQXBO33BCQL4W5L5RKJAYACZHYJMHCX6XORWSWQ, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.038582503 (376h31m10.809223838s) [noise=true , address=104.36.82.2:4730 , piece_id=EVW56CTEQD7BPUSYS2U5NXO4VRG6YZVEITV6OU5XKT3AU3MRWRYQ, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.038924036 (376h31m10.809565371s) [noise=true , address=104.36.82.8:4969 , piece_id=6IKLPKOPP2HBAOC5FAI5IE5SNW2B2UWNWT3Q6S4WWKGDLRCARVWA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.039254127 (376h31m10.809895453s) [noise=true , address=77.90.31.21:4878 , piece_id=4YNDQENFOKMHFJFVWWBC7UK4TQP745OUW7S2P43BICF2SNFAWIHA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.039603647 (376h31m10.810244982s) [noise=true , address=45.91.116.12:4227 , piece_id=23YQLIAPNPNMSA3LL67PNYCRDMWJ4GXZY3VF354XXUH5YEXFWVUA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:37.680432693 (376h31m12.451074028s) [upload=301225]: Next returned nil segment
Jul 9 01:41:37.680461809 (376h31m12.451103144s) [upload=301225]: Waiting for error group managing segments...
Jul 9 01:41:38.011139865 (376h31m12.7817812s) [noise=true , address=104.36.82.16:4294 , piece_id=CE4MNKGLLBYQAQ7KGVTTGUMWRTD2NTWF55NOPILQC5YPXDQT3Z6A, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.014224286 (376h31m12.784865621s) [noise=true , address=104.36.82.9:4007 , piece_id=TMZJKWZDZHE4ANML45KWGRPDG5UOTNIYUZFDH3LJYZHMU6LBNTIA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.116297430 (376h31m12.886938765s) [noise=true , address=45.91.116.6:4975 , piece_id=LDM5B2S4NP43AXC3DJXTUM66Z6EM3SBKPP6JHTMLA7RZATZQ4QQA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.118333195 (376h31m12.88897453s) [noise=true , address=45.91.116.13:4153 , piece_id=UMTA7DVZBXEYRQQXBO33BCQL4W5L5RKJAYACZHYJMHCX6XORWSWQ, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.130186111 (376h31m12.900827446s) [noise=true , address=45.91.116.8:4045 , piece_id=Z37SMFGUPVJVP6RNGUIDH6Q456OJFVZ7SNFFYT3NT3AOXVYR5TXA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.130610163 (376h31m12.901251498s) [noise=true , address=45.91.116.10:4082 , piece_id=DNZJWYESTZZY2QVM2SRD32AS6GDWSSO7ODNWMHBXJAGRNQ3FMSMQ, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.171163518 (376h31m12.941804854s) [noise=true , address=77.90.31.19:4843 , piece_id=JH6TQLWG4LCPENFHL6ORQQBR5YDUOU2QAY7OLQ5KYXOK2DW5DRGA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.187728911 (376h31m12.958370246s) [noise=true , address=77.90.31.12:4592 , piece_id=J7LR65VEHL5D7XBO2HMEMFZVFBPEES5IL7FQTV5IJK4ZHEHGDKOA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.190933191 (376h31m12.961574526s) [noise=true , address=77.90.31.21:4878 , piece_id=4YNDQENFOKMHFJFVWWBC7UK4TQP745OUW7S2P43BICF2SNFAWIHA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.218675778 (376h31m12.989317113s) [noise=true , address=45.91.116.12:4227 , piece_id=23YQLIAPNPNMSA3LL67PNYCRDMWJ4GXZY3VF354XXUH5YEXFWVUA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.225536314 (376h31m12.996177649s) [noise=true , address=45.91.116.14:4190 , piece_id=CZGKNHBIJP7HZIC65I6ESRFRQLBKTEAQJVVUQHHZHFO7TNNROJ4A, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:42:05.141496679 (376h31m39.912138074s) [noise=true , address=104.36.82.8:4969 , piece_id=6IKLPKOPP2HBAOC5FAI5IE5SNW2B2UWNWT3Q6S4WWKGDLRCARVWA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 01:42:05.141543177 (376h31m39.912184512s) [noise=true , address=104.36.82.14:4222 , piece_id=FU4BHM26LXD5XUTT333YO7ETORBXWP6QWCLUUJI2MBMPWLEFMGAA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 188.166.225.170:60278->104.36.82.14:4222: use of closed network connection; write tcp 188.166.225.170:60278->104.36.82.14:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60278->104.36.82.14:4222: use of closed network connection
Jul 9 01:42:05.141602951 (376h31m39.912244286s) [noise=true , address=104.36.82.5:4840 , piece_id=QKGEMZVR5P3B6ARIGS4VQDSWGAOVC5DU65Y3VLXUY2MAYVKDUQCQ, seg_pos={0 0}, upload=301225]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: write tcp 188.166.225.170:35250->104.36.82.5:4840: use of closed network connection; write tcp 188.166.225.170:35250->104.36.82.5:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35250->104.36.82.5:4840: use of closed network connection
Jul 9 01:42:05.141829285 (376h31m39.91247062s) [noise=true , address=104.36.82.11:4114 , piece_id=5ISRL5WJST2SCUOPBB7KX53T3PKO2CH5X7QKCPGSKSDO3KA5BYHQ, seg_pos={0 0}, upload=301225]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 188.166.225.170:54484->104.36.82.11:4114: use of closed network connection; write tcp 188.166.225.170:54484->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54484->104.36.82.11:4114: use of closed network connection
Jul 9 01:42:05.141852700 (376h31m39.912494025s) [noise=true , address=104.36.82.2:4730 , piece_id=EVW56CTEQD7BPUSYS2U5NXO4VRG6YZVEITV6OU5XKT3AU3MRWRYQ, seg_pos={0 0}, upload=301225]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 188.166.225.170:59748->104.36.82.2:4730: use of closed network connection; write tcp 188.166.225.170:59748->104.36.82.2:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59748->104.36.82.2:4730: use of closed network connection
Jul 9 01:42:05.141879792 (376h31m39.912521127s) [seg_pos={0 0}, upload=301225]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 9 01:42:05.366924712 (376h31m40.137566037s) [upload=301225]: Done uploading segments...
Jul 9 01:42:05.366949890 (376h31m40.137591225s) [upload=301225]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 5.75863408s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [.......X..........................X......X..........X.......] duration:
29.815347424s
FAIL (2025-07-09 01:50:59 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
s) [upload=301248]: Got next segment. Inline: false
Jul 9 01:43:24.322645491 (376h32m59.093286826s) [upload=301248]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 01:43:24.322650441 (376h32m59.093291777s) [upload=301248]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 01:43:24.769774209 (376h32m59.540415544s) [seg_pos={0 0}, upload=301248]: Begin upload segment...
Jul 9 01:43:24.769830437 (376h32m59.540471772s) [seg_pos={0 0}, upload=301248]: Done begin upload segment.
Jul 9 01:43:24.769852048 (376h32m59.540493373s) [noise=true , address=79.127.205.234:4614 , piece_id=JPPMN7KOKXFYF2YDK4RC7AU6KYV2UHQELFEG6ZZ6W5HQ2MKFVN5A, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.769864432 (376h32m59.540505767s) [noise=true , address=45.91.116.11:4123 , piece_id=WSMWN6I2QVBTO6OCRRDCMT3G3TESW3H2ZET5YNWFUPVENPN3UWKQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.770250630 (376h32m59.540891965s) [noise=true , address=104.36.82.22:4511 , piece_id=3AIKVWFRA4CLMPRDYFNX6EA7XO22MB7RQEOVNXPUVUCNFAHPSFCQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.770361543 (376h32m59.541002878s) [noise=true , address=77.90.31.21:4886 , piece_id=YRDZYEV6KU5YISF4DIS3GTZVBFRCZEOIW2DG7VCKHO26NRXMHMVQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.770542389 (376h32m59.541183724s) [noise=true , address=77.90.31.19:4848 , piece_id=RV7TBRZW2YH64X5CLXFHFTJD2WQCEM3P7OW3NYZ2IWC2E3DJDG5A, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.770626239 (376h32m59.541267594s) [noise=true , address=104.36.82.19:4411 , piece_id=R27NIJUXHNTQWK4IXXSVS2QC535JAZJ3K2WV4VANVOGKRYV7YADA, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.770818237 (376h32m59.541459572s) [noise=true , address=104.36.82.40:4955 , piece_id=3TVDS5QR5267W6VKWNXOPW6ZSTXISGXFBIPKK7E3S6ML7C4X4MUQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.770918087 (376h32m59.541559422s) [noise=true , address=104.36.82.12:4160 , piece_id=2Z3FQL5GTT26HC2GPKO5YPOLPB2Z75K4UG3FDPMOEDCCD6Q6MULQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.771127618 (376h32m59.541768953s) [noise=true , address=77.90.31.14:4677 , piece_id=Z7SGXIO722E3U7PWA4AWOUD5T27VSQUCG3ZBVP22WJRFSRKOEXAQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.771146836 (376h32m59.541788171s) [noise=true , address=77.90.31.12:4622 , piece_id=25K7AOPYFLZ6EDAA3SPHHE6S635PN5VB3ZK4LIULBHKQGYYQ3C7A, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.771367858 (376h32m59.542009193s) [noise=true , address=77.90.31.18:4825 , piece_id=NFOYMUTMKMKTPCAEPBTWL43ISOETOAM4WKVAUNHCETNW75IQLLMQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.771421060 (376h32m59.542062395s) [noise=true , address=77.90.31.14:4691 , piece_id=ELMXSLQT6TNOMGNUGM4BEHZ2P7OZWNSX6EOCHDAYOD7OIANFUVHA, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.771586527 (376h32m59.542227852s) [noise=true , address=64.49.13.169:4706 , piece_id=2CKVJCKZJH4F3VC3X7XJOWMYHVHU52POGAA7VMAU4YM6HTVHC5PA, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.771699443 (376h32m59.542340778s) [noise=true , address=104.36.82.44:4898 , piece_id=X47T6NYNODFSVKHJGWIH2WJT7BA3UVP7BEBYVKPPQBMN4QOQCJ6Q, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.771818129 (376h32m59.542459465s) [noise=true , address=104.36.82.34:4551 , piece_id=HWYKZWYEKMJT3JHDPHRWIAFKP5NZQ72OGR4SPR2I2SRRABLA663A, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.772055795 (376h32m59.54269713s) [noise=true , address=77.90.31.10:4543 , piece_id=QCKPGFCSQ3N57BWXMOSW5LAVA4KOI3EVUFU2OMPN7CMAOYAH4RWQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.772089298 (376h32m59.542730623s) [noise=true , address=104.36.82.23:4459 , piece_id=SOG4BGCMVK36OVMKHGIYGSVGX2ZZ2PWZV74S2LQS6IZQRGDYJVYA, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.772274673 (376h32m59.542916008s) [noise=true , address=77.90.31.15:4715 , piece_id=ITPPGCX32ZLU72XXE7RG6AYE3PM7IILVNPXM5A4NJ2S7P6U4Y2FA, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:25.540057386 (376h33m0.310698721s) [noise=true , address=64.49.13.169:4706 , piece_id=2CKVJCKZJH4F3VC3X7XJOWMYHVHU52POGAA7VMAU4YM6HTVHC5PA, seg_pos={0 0}, upload=301248]: Done uploading piece. err: ecclient: upload failed (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7, address:64.49.13.169:4706): protocol: storage node overloaded, request limit: 2000
Jul 9 01:43:25.878055136 (376h33m0.648696481s) [upload=301248]: Next returned nil segment
Jul 9 01:43:25.878073130 (376h33m0.648714465s) [upload=301248]: Waiting for error group managing segments...
Jul 9 01:43:26.175673001 (376h33m0.946314336s) [noise=true , address=79.127.205.234:4614 , piece_id=JPPMN7KOKXFYF2YDK4RC7AU6KYV2UHQELFEG6ZZ6W5HQ2MKFVN5A, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.774276626 (376h33m1.544917961s) [noise=true , address=77.90.31.12:4622 , piece_id=25K7AOPYFLZ6EDAA3SPHHE6S635PN5VB3ZK4LIULBHKQGYYQ3C7A, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.778019557 (376h33m1.548660892s) [noise=true , address=77.90.31.21:4886 , piece_id=YRDZYEV6KU5YISF4DIS3GTZVBFRCZEOIW2DG7VCKHO26NRXMHMVQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.779099603 (376h33m1.549740938s) [noise=true , address=77.90.31.14:4691 , piece_id=ELMXSLQT6TNOMGNUGM4BEHZ2P7OZWNSX6EOCHDAYOD7OIANFUVHA, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.779161091 (376h33m1.549802426s) [noise=true , address=77.90.31.18:4825 , piece_id=NFOYMUTMKMKTPCAEPBTWL43ISOETOAM4WKVAUNHCETNW75IQLLMQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.780045593 (376h33m1.550686928s) [noise=true , address=77.90.31.14:4677 , piece_id=Z7SGXIO722E3U7PWA4AWOUD5T27VSQUCG3ZBVP22WJRFSRKOEXAQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.795740620 (376h33m1.566381976s) [noise=true , address=77.90.31.10:4543 , piece_id=QCKPGFCSQ3N57BWXMOSW5LAVA4KOI3EVUFU2OMPN7CMAOYAH4RWQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.795758073 (376h33m1.566399408s) [noise=true , address=77.90.31.15:4715 , piece_id=ITPPGCX32ZLU72XXE7RG6AYE3PM7IILVNPXM5A4NJ2S7P6U4Y2FA, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.812585147 (376h33m1.583226483s) [noise=true , address=77.90.31.19:4848 , piece_id=RV7TBRZW2YH64X5CLXFHFTJD2WQCEM3P7OW3NYZ2IWC2E3DJDG5A, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:27.386630478 (376h33m2.157271803s) [noise=true , address=45.91.116.11:4123 , piece_id=WSMWN6I2QVBTO6OCRRDCMT3G3TESW3H2ZET5YNWFUPVENPN3UWKQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:37.603914421 (376h33m12.374555756s) [noise=true , address=104.36.82.34:4551 , piece_id=HWYKZWYEKMJT3JHDPHRWIAFKP5NZQ72OGR4SPR2I2SRRABLA663A, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:41.629246242 (376h33m16.399887567s) [noise=true , address=104.36.82.12:4160 , piece_id=2Z3FQL5GTT26HC2GPKO5YPOLPB2Z75K4UG3FDPMOEDCCD6Q6MULQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:44:10.222815170 (376h33m44.993456505s) [noise=true , address=104.36.82.22:4511 , piece_id=3AIKVWFRA4CLMPRDYFNX6EA7XO22MB7RQEOVNXPUVUCNFAHPSFCQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:44:21.573027994 (376h33m56.343669329s) [noise=true , address=104.36.82.23:4459 , piece_id=SOG4BGCMVK36OVMKHGIYGSVGX2ZZ2PWZV74S2LQS6IZQRGDYJVYA, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:44:21.573054295 (376h33m56.34369563s) [seg_pos={0 0}, upload=301248]: Segment reached optimal threshold of 14 pieces.
Jul 9 01:44:21.573140269 (376h33m56.343781604s) [noise=true , address=104.36.82.40:4955 , piece_id=3TVDS5QR5267W6VKWNXOPW6ZSTXISGXFBIPKK7E3S6ML7C4X4MUQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ssLq5bo7og4z8ZPd28f2H6WUExsWohBBZUnciGaH5KKpcQ25d): protocol: expected piece hash; context canceled; EOF
Jul 9 01:44:21.573153795 (376h33m56.34379513s) [noise=true , address=104.36.82.19:4411 , piece_id=R27NIJUXHNTQWK4IXXSVS2QC535JAZJ3K2WV4VANVOGKRYV7YADA, seg_pos={0 0}, upload=301248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nradx4SMYLkuJw3Ad6o9Sq1iNtQR6GDu8qg7tCzZzNcTjVfyd): protocol: expected piece hash; context canceled; EOF
Jul 9 01:44:21.573259487 (376h33m56.343900822s) [noise=true , address=104.36.82.44:4898 , piece_id=X47T6NYNODFSVKHJGWIH2WJT7BA3UVP7BEBYVKPPQBMN4QOQCJ6Q, seg_pos={0 0}, upload=301248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc): protocol: write tcp 188.166.225.170:40614->104.36.82.44:4898: use of closed network connection; write tcp 188.166.225.170:40614->104.36.82.44:4898: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40614->104.36.82.44:4898: use of closed network connection
Jul 9 01:44:21.573276430 (376h33m56.343917755s) [seg_pos={0 0}, upload=301248]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 01:44:21.573349429 (376h33m56.343990764s) [upload=301248]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 01:44:21.573353537 (376h33m56.343994873s) [upload=301248]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 01:44:21.865577192 (376h33m56.636218527s) [upload=301248]: Done uploading segments...
Jul 9 01:44:21.865594906 (376h33m56.636236232s) [upload=301248]: Upload finished. err: <nil>
Duration: 29.815382009s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [X....X...X.X....X...X.X..X...X..X............X......X..X...X] duration:
30.001456396s
FAIL (2025-07-09 01:50:41 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001484991s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [X........................X......................X...........] duration:
5.756317677s
FAIL (2025-07-09 01:50:31 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Jul 9 01:41:35.784016383 (376h31m10.554657718s) [upload=301225]: Starting upload
Jul 9 01:41:35.784073732 (376h31m10.554715067s) [upload=301225]: Done starting upload
Jul 9 01:41:35.784180246 (376h31m10.554821571s) [upload=301225]: Uploading segments...
Jul 9 01:41:35.784275688 (376h31m10.554917023s) [upload=301225]: Got next segment. Inline: false
Jul 9 01:41:35.784309523 (376h31m10.554950848s) [upload=301225]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 01:41:35.784314523 (376h31m10.554955858s) [upload=301225]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 01:41:36.035706020 (376h31m10.806347375s) [seg_pos={0 0}, upload=301225]: Begin upload segment...
Jul 9 01:41:36.035755273 (376h31m10.806396598s) [seg_pos={0 0}, upload=301225]: Done begin upload segment.
Jul 9 01:41:36.035768209 (376h31m10.806409574s) [noise=true , address=45.91.116.14:4190 , piece_id=CZGKNHBIJP7HZIC65I6ESRFRQLBKTEAQJVVUQHHZHFO7TNNROJ4A, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.035782255 (376h31m10.80642359s) [noise=true , address=104.36.82.16:4294 , piece_id=CE4MNKGLLBYQAQ7KGVTTGUMWRTD2NTWF55NOPILQC5YPXDQT3Z6A, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.035859012 (376h31m10.806500367s) [noise=true , address=104.36.82.5:4840 , piece_id=QKGEMZVR5P3B6ARIGS4VQDSWGAOVC5DU65Y3VLXUY2MAYVKDUQCQ, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.036404877 (376h31m10.807046232s) [noise=true , address=45.91.116.8:4045 , piece_id=Z37SMFGUPVJVP6RNGUIDH6Q456OJFVZ7SNFFYT3NT3AOXVYR5TXA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.036420547 (376h31m10.807061892s) [noise=true , address=77.90.31.19:4843 , piece_id=JH6TQLWG4LCPENFHL6ORQQBR5YDUOU2QAY7OLQ5KYXOK2DW5DRGA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.036942675 (376h31m10.80758403s) [noise=true , address=104.36.82.9:4007 , piece_id=TMZJKWZDZHE4ANML45KWGRPDG5UOTNIYUZFDH3LJYZHMU6LBNTIA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.037006067 (376h31m10.807647412s) [noise=true , address=45.91.116.10:4082 , piece_id=DNZJWYESTZZY2QVM2SRD32AS6GDWSSO7ODNWMHBXJAGRNQ3FMSMQ, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.037346698 (376h31m10.807988043s) [noise=true , address=45.91.116.6:4975 , piece_id=LDM5B2S4NP43AXC3DJXTUM66Z6EM3SBKPP6JHTMLA7RZATZQ4QQA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.037776601 (376h31m10.808417936s) [noise=true , address=104.36.82.11:4114 , piece_id=5ISRL5WJST2SCUOPBB7KX53T3PKO2CH5X7QKCPGSKSDO3KA5BYHQ, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.038177858 (376h31m10.808819193s) [noise=true , address=104.36.82.14:4222 , piece_id=FU4BHM26LXD5XUTT333YO7ETORBXWP6QWCLUUJI2MBMPWLEFMGAA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.038258122 (376h31m10.808899467s) [noise=true , address=77.90.31.12:4592 , piece_id=J7LR65VEHL5D7XBO2HMEMFZVFBPEES5IL7FQTV5IJK4ZHEHGDKOA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.038562023 (376h31m10.809203359s) [noise=true , address=45.91.116.13:4153 , piece_id=UMTA7DVZBXEYRQQXBO33BCQL4W5L5RKJAYACZHYJMHCX6XORWSWQ, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.038582503 (376h31m10.809223838s) [noise=true , address=104.36.82.2:4730 , piece_id=EVW56CTEQD7BPUSYS2U5NXO4VRG6YZVEITV6OU5XKT3AU3MRWRYQ, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.038924036 (376h31m10.809565371s) [noise=true , address=104.36.82.8:4969 , piece_id=6IKLPKOPP2HBAOC5FAI5IE5SNW2B2UWNWT3Q6S4WWKGDLRCARVWA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.039254127 (376h31m10.809895453s) [noise=true , address=77.90.31.21:4878 , piece_id=4YNDQENFOKMHFJFVWWBC7UK4TQP745OUW7S2P43BICF2SNFAWIHA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.039603647 (376h31m10.810244982s) [noise=true , address=45.91.116.12:4227 , piece_id=23YQLIAPNPNMSA3LL67PNYCRDMWJ4GXZY3VF354XXUH5YEXFWVUA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:37.680432693 (376h31m12.451074028s) [upload=301225]: Next returned nil segment
Jul 9 01:41:37.680461809 (376h31m12.451103144s) [upload=301225]: Waiting for error group managing segments...
Jul 9 01:41:38.011139865 (376h31m12.7817812s) [noise=true , address=104.36.82.16:4294 , piece_id=CE4MNKGLLBYQAQ7KGVTTGUMWRTD2NTWF55NOPILQC5YPXDQT3Z6A, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.014224286 (376h31m12.784865621s) [noise=true , address=104.36.82.9:4007 , piece_id=TMZJKWZDZHE4ANML45KWGRPDG5UOTNIYUZFDH3LJYZHMU6LBNTIA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.116297430 (376h31m12.886938765s) [noise=true , address=45.91.116.6:4975 , piece_id=LDM5B2S4NP43AXC3DJXTUM66Z6EM3SBKPP6JHTMLA7RZATZQ4QQA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.118333195 (376h31m12.88897453s) [noise=true , address=45.91.116.13:4153 , piece_id=UMTA7DVZBXEYRQQXBO33BCQL4W5L5RKJAYACZHYJMHCX6XORWSWQ, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.130186111 (376h31m12.900827446s) [noise=true , address=45.91.116.8:4045 , piece_id=Z37SMFGUPVJVP6RNGUIDH6Q456OJFVZ7SNFFYT3NT3AOXVYR5TXA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.130610163 (376h31m12.901251498s) [noise=true , address=45.91.116.10:4082 , piece_id=DNZJWYESTZZY2QVM2SRD32AS6GDWSSO7ODNWMHBXJAGRNQ3FMSMQ, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.171163518 (376h31m12.941804854s) [noise=true , address=77.90.31.19:4843 , piece_id=JH6TQLWG4LCPENFHL6ORQQBR5YDUOU2QAY7OLQ5KYXOK2DW5DRGA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.187728911 (376h31m12.958370246s) [noise=true , address=77.90.31.12:4592 , piece_id=J7LR65VEHL5D7XBO2HMEMFZVFBPEES5IL7FQTV5IJK4ZHEHGDKOA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.190933191 (376h31m12.961574526s) [noise=true , address=77.90.31.21:4878 , piece_id=4YNDQENFOKMHFJFVWWBC7UK4TQP745OUW7S2P43BICF2SNFAWIHA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.218675778 (376h31m12.989317113s) [noise=true , address=45.91.116.12:4227 , piece_id=23YQLIAPNPNMSA3LL67PNYCRDMWJ4GXZY3VF354XXUH5YEXFWVUA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.225536314 (376h31m12.996177649s) [noise=true , address=45.91.116.14:4190 , piece_id=CZGKNHBIJP7HZIC65I6ESRFRQLBKTEAQJVVUQHHZHFO7TNNROJ4A, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:42:05.141496679 (376h31m39.912138074s) [noise=true , address=104.36.82.8:4969 , piece_id=6IKLPKOPP2HBAOC5FAI5IE5SNW2B2UWNWT3Q6S4WWKGDLRCARVWA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 01:42:05.141543177 (376h31m39.912184512s) [noise=true , address=104.36.82.14:4222 , piece_id=FU4BHM26LXD5XUTT333YO7ETORBXWP6QWCLUUJI2MBMPWLEFMGAA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 188.166.225.170:60278->104.36.82.14:4222: use of closed network connection; write tcp 188.166.225.170:60278->104.36.82.14:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60278->104.36.82.14:4222: use of closed network connection
Jul 9 01:42:05.141602951 (376h31m39.912244286s) [noise=true , address=104.36.82.5:4840 , piece_id=QKGEMZVR5P3B6ARIGS4VQDSWGAOVC5DU65Y3VLXUY2MAYVKDUQCQ, seg_pos={0 0}, upload=301225]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: write tcp 188.166.225.170:35250->104.36.82.5:4840: use of closed network connection; write tcp 188.166.225.170:35250->104.36.82.5:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35250->104.36.82.5:4840: use of closed network connection
Jul 9 01:42:05.141829285 (376h31m39.91247062s) [noise=true , address=104.36.82.11:4114 , piece_id=5ISRL5WJST2SCUOPBB7KX53T3PKO2CH5X7QKCPGSKSDO3KA5BYHQ, seg_pos={0 0}, upload=301225]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 188.166.225.170:54484->104.36.82.11:4114: use of closed network connection; write tcp 188.166.225.170:54484->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54484->104.36.82.11:4114: use of closed network connection
Jul 9 01:42:05.141852700 (376h31m39.912494025s) [noise=true , address=104.36.82.2:4730 , piece_id=EVW56CTEQD7BPUSYS2U5NXO4VRG6YZVEITV6OU5XKT3AU3MRWRYQ, seg_pos={0 0}, upload=301225]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 188.166.225.170:59748->104.36.82.2:4730: use of closed network connection; write tcp 188.166.225.170:59748->104.36.82.2:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59748->104.36.82.2:4730: use of closed network connection
Jul 9 01:42:05.141879792 (376h31m39.912521127s) [seg_pos={0 0}, upload=301225]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 9 01:42:05.366924712 (376h31m40.137566037s) [upload=301225]: Done uploading segments...
Jul 9 01:42:05.366949890 (376h31m40.137591225s) [upload=301225]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 5.756359596s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [........X..........................X......X..........X......] duration:
34.186289983s
FAIL (2025-07-09 01:50:18 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
s) [upload=301248]: Got next segment. Inline: false
Jul 9 01:43:24.322645491 (376h32m59.093286826s) [upload=301248]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 01:43:24.322650441 (376h32m59.093291777s) [upload=301248]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 01:43:24.769774209 (376h32m59.540415544s) [seg_pos={0 0}, upload=301248]: Begin upload segment...
Jul 9 01:43:24.769830437 (376h32m59.540471772s) [seg_pos={0 0}, upload=301248]: Done begin upload segment.
Jul 9 01:43:24.769852048 (376h32m59.540493373s) [noise=true , address=79.127.205.234:4614 , piece_id=JPPMN7KOKXFYF2YDK4RC7AU6KYV2UHQELFEG6ZZ6W5HQ2MKFVN5A, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.769864432 (376h32m59.540505767s) [noise=true , address=45.91.116.11:4123 , piece_id=WSMWN6I2QVBTO6OCRRDCMT3G3TESW3H2ZET5YNWFUPVENPN3UWKQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.770250630 (376h32m59.540891965s) [noise=true , address=104.36.82.22:4511 , piece_id=3AIKVWFRA4CLMPRDYFNX6EA7XO22MB7RQEOVNXPUVUCNFAHPSFCQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.770361543 (376h32m59.541002878s) [noise=true , address=77.90.31.21:4886 , piece_id=YRDZYEV6KU5YISF4DIS3GTZVBFRCZEOIW2DG7VCKHO26NRXMHMVQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.770542389 (376h32m59.541183724s) [noise=true , address=77.90.31.19:4848 , piece_id=RV7TBRZW2YH64X5CLXFHFTJD2WQCEM3P7OW3NYZ2IWC2E3DJDG5A, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.770626239 (376h32m59.541267594s) [noise=true , address=104.36.82.19:4411 , piece_id=R27NIJUXHNTQWK4IXXSVS2QC535JAZJ3K2WV4VANVOGKRYV7YADA, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.770818237 (376h32m59.541459572s) [noise=true , address=104.36.82.40:4955 , piece_id=3TVDS5QR5267W6VKWNXOPW6ZSTXISGXFBIPKK7E3S6ML7C4X4MUQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.770918087 (376h32m59.541559422s) [noise=true , address=104.36.82.12:4160 , piece_id=2Z3FQL5GTT26HC2GPKO5YPOLPB2Z75K4UG3FDPMOEDCCD6Q6MULQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.771127618 (376h32m59.541768953s) [noise=true , address=77.90.31.14:4677 , piece_id=Z7SGXIO722E3U7PWA4AWOUD5T27VSQUCG3ZBVP22WJRFSRKOEXAQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.771146836 (376h32m59.541788171s) [noise=true , address=77.90.31.12:4622 , piece_id=25K7AOPYFLZ6EDAA3SPHHE6S635PN5VB3ZK4LIULBHKQGYYQ3C7A, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.771367858 (376h32m59.542009193s) [noise=true , address=77.90.31.18:4825 , piece_id=NFOYMUTMKMKTPCAEPBTWL43ISOETOAM4WKVAUNHCETNW75IQLLMQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.771421060 (376h32m59.542062395s) [noise=true , address=77.90.31.14:4691 , piece_id=ELMXSLQT6TNOMGNUGM4BEHZ2P7OZWNSX6EOCHDAYOD7OIANFUVHA, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.771586527 (376h32m59.542227852s) [noise=true , address=64.49.13.169:4706 , piece_id=2CKVJCKZJH4F3VC3X7XJOWMYHVHU52POGAA7VMAU4YM6HTVHC5PA, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.771699443 (376h32m59.542340778s) [noise=true , address=104.36.82.44:4898 , piece_id=X47T6NYNODFSVKHJGWIH2WJT7BA3UVP7BEBYVKPPQBMN4QOQCJ6Q, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.771818129 (376h32m59.542459465s) [noise=true , address=104.36.82.34:4551 , piece_id=HWYKZWYEKMJT3JHDPHRWIAFKP5NZQ72OGR4SPR2I2SRRABLA663A, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.772055795 (376h32m59.54269713s) [noise=true , address=77.90.31.10:4543 , piece_id=QCKPGFCSQ3N57BWXMOSW5LAVA4KOI3EVUFU2OMPN7CMAOYAH4RWQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.772089298 (376h32m59.542730623s) [noise=true , address=104.36.82.23:4459 , piece_id=SOG4BGCMVK36OVMKHGIYGSVGX2ZZ2PWZV74S2LQS6IZQRGDYJVYA, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.772274673 (376h32m59.542916008s) [noise=true , address=77.90.31.15:4715 , piece_id=ITPPGCX32ZLU72XXE7RG6AYE3PM7IILVNPXM5A4NJ2S7P6U4Y2FA, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:25.540057386 (376h33m0.310698721s) [noise=true , address=64.49.13.169:4706 , piece_id=2CKVJCKZJH4F3VC3X7XJOWMYHVHU52POGAA7VMAU4YM6HTVHC5PA, seg_pos={0 0}, upload=301248]: Done uploading piece. err: ecclient: upload failed (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7, address:64.49.13.169:4706): protocol: storage node overloaded, request limit: 2000
Jul 9 01:43:25.878055136 (376h33m0.648696481s) [upload=301248]: Next returned nil segment
Jul 9 01:43:25.878073130 (376h33m0.648714465s) [upload=301248]: Waiting for error group managing segments...
Jul 9 01:43:26.175673001 (376h33m0.946314336s) [noise=true , address=79.127.205.234:4614 , piece_id=JPPMN7KOKXFYF2YDK4RC7AU6KYV2UHQELFEG6ZZ6W5HQ2MKFVN5A, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.774276626 (376h33m1.544917961s) [noise=true , address=77.90.31.12:4622 , piece_id=25K7AOPYFLZ6EDAA3SPHHE6S635PN5VB3ZK4LIULBHKQGYYQ3C7A, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.778019557 (376h33m1.548660892s) [noise=true , address=77.90.31.21:4886 , piece_id=YRDZYEV6KU5YISF4DIS3GTZVBFRCZEOIW2DG7VCKHO26NRXMHMVQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.779099603 (376h33m1.549740938s) [noise=true , address=77.90.31.14:4691 , piece_id=ELMXSLQT6TNOMGNUGM4BEHZ2P7OZWNSX6EOCHDAYOD7OIANFUVHA, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.779161091 (376h33m1.549802426s) [noise=true , address=77.90.31.18:4825 , piece_id=NFOYMUTMKMKTPCAEPBTWL43ISOETOAM4WKVAUNHCETNW75IQLLMQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.780045593 (376h33m1.550686928s) [noise=true , address=77.90.31.14:4677 , piece_id=Z7SGXIO722E3U7PWA4AWOUD5T27VSQUCG3ZBVP22WJRFSRKOEXAQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.795740620 (376h33m1.566381976s) [noise=true , address=77.90.31.10:4543 , piece_id=QCKPGFCSQ3N57BWXMOSW5LAVA4KOI3EVUFU2OMPN7CMAOYAH4RWQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.795758073 (376h33m1.566399408s) [noise=true , address=77.90.31.15:4715 , piece_id=ITPPGCX32ZLU72XXE7RG6AYE3PM7IILVNPXM5A4NJ2S7P6U4Y2FA, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.812585147 (376h33m1.583226483s) [noise=true , address=77.90.31.19:4848 , piece_id=RV7TBRZW2YH64X5CLXFHFTJD2WQCEM3P7OW3NYZ2IWC2E3DJDG5A, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:27.386630478 (376h33m2.157271803s) [noise=true , address=45.91.116.11:4123 , piece_id=WSMWN6I2QVBTO6OCRRDCMT3G3TESW3H2ZET5YNWFUPVENPN3UWKQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:37.603914421 (376h33m12.374555756s) [noise=true , address=104.36.82.34:4551 , piece_id=HWYKZWYEKMJT3JHDPHRWIAFKP5NZQ72OGR4SPR2I2SRRABLA663A, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:41.629246242 (376h33m16.399887567s) [noise=true , address=104.36.82.12:4160 , piece_id=2Z3FQL5GTT26HC2GPKO5YPOLPB2Z75K4UG3FDPMOEDCCD6Q6MULQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:44:10.222815170 (376h33m44.993456505s) [noise=true , address=104.36.82.22:4511 , piece_id=3AIKVWFRA4CLMPRDYFNX6EA7XO22MB7RQEOVNXPUVUCNFAHPSFCQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:44:21.573027994 (376h33m56.343669329s) [noise=true , address=104.36.82.23:4459 , piece_id=SOG4BGCMVK36OVMKHGIYGSVGX2ZZ2PWZV74S2LQS6IZQRGDYJVYA, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:44:21.573054295 (376h33m56.34369563s) [seg_pos={0 0}, upload=301248]: Segment reached optimal threshold of 14 pieces.
Jul 9 01:44:21.573140269 (376h33m56.343781604s) [noise=true , address=104.36.82.40:4955 , piece_id=3TVDS5QR5267W6VKWNXOPW6ZSTXISGXFBIPKK7E3S6ML7C4X4MUQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ssLq5bo7og4z8ZPd28f2H6WUExsWohBBZUnciGaH5KKpcQ25d): protocol: expected piece hash; context canceled; EOF
Jul 9 01:44:21.573153795 (376h33m56.34379513s) [noise=true , address=104.36.82.19:4411 , piece_id=R27NIJUXHNTQWK4IXXSVS2QC535JAZJ3K2WV4VANVOGKRYV7YADA, seg_pos={0 0}, upload=301248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nradx4SMYLkuJw3Ad6o9Sq1iNtQR6GDu8qg7tCzZzNcTjVfyd): protocol: expected piece hash; context canceled; EOF
Jul 9 01:44:21.573259487 (376h33m56.343900822s) [noise=true , address=104.36.82.44:4898 , piece_id=X47T6NYNODFSVKHJGWIH2WJT7BA3UVP7BEBYVKPPQBMN4QOQCJ6Q, seg_pos={0 0}, upload=301248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc): protocol: write tcp 188.166.225.170:40614->104.36.82.44:4898: use of closed network connection; write tcp 188.166.225.170:40614->104.36.82.44:4898: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40614->104.36.82.44:4898: use of closed network connection
Jul 9 01:44:21.573276430 (376h33m56.343917755s) [seg_pos={0 0}, upload=301248]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 01:44:21.573349429 (376h33m56.343990764s) [upload=301248]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 01:44:21.573353537 (376h33m56.343994873s) [upload=301248]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 01:44:21.865577192 (376h33m56.636218527s) [upload=301248]: Done uploading segments...
Jul 9 01:44:21.865594906 (376h33m56.636236232s) [upload=301248]: Upload finished. err: <nil>
Duration: 34.18631392s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [XX....X...X.X....X...X.X..X...X..X............X......X..X...] duration:
20.990955398s
FAIL (2025-07-09 01:50:04 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 20.990977622s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
13 out of 60 is failed (threshold=3) [.....................X......XX.XX..X......XXX....XX.....XX..] duration:
42.890237256s
FAIL (2025-07-09 01:49:54 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 42.890255501s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.X........................X......................X..........] duration:
5.938388789s
FAIL (2025-07-09 01:49:21 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Jul 9 01:41:35.784016383 (376h31m10.554657718s) [upload=301225]: Starting upload
Jul 9 01:41:35.784073732 (376h31m10.554715067s) [upload=301225]: Done starting upload
Jul 9 01:41:35.784180246 (376h31m10.554821571s) [upload=301225]: Uploading segments...
Jul 9 01:41:35.784275688 (376h31m10.554917023s) [upload=301225]: Got next segment. Inline: false
Jul 9 01:41:35.784309523 (376h31m10.554950848s) [upload=301225]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 01:41:35.784314523 (376h31m10.554955858s) [upload=301225]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 01:41:36.035706020 (376h31m10.806347375s) [seg_pos={0 0}, upload=301225]: Begin upload segment...
Jul 9 01:41:36.035755273 (376h31m10.806396598s) [seg_pos={0 0}, upload=301225]: Done begin upload segment.
Jul 9 01:41:36.035768209 (376h31m10.806409574s) [noise=true , address=45.91.116.14:4190 , piece_id=CZGKNHBIJP7HZIC65I6ESRFRQLBKTEAQJVVUQHHZHFO7TNNROJ4A, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.035782255 (376h31m10.80642359s) [noise=true , address=104.36.82.16:4294 , piece_id=CE4MNKGLLBYQAQ7KGVTTGUMWRTD2NTWF55NOPILQC5YPXDQT3Z6A, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.035859012 (376h31m10.806500367s) [noise=true , address=104.36.82.5:4840 , piece_id=QKGEMZVR5P3B6ARIGS4VQDSWGAOVC5DU65Y3VLXUY2MAYVKDUQCQ, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.036404877 (376h31m10.807046232s) [noise=true , address=45.91.116.8:4045 , piece_id=Z37SMFGUPVJVP6RNGUIDH6Q456OJFVZ7SNFFYT3NT3AOXVYR5TXA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.036420547 (376h31m10.807061892s) [noise=true , address=77.90.31.19:4843 , piece_id=JH6TQLWG4LCPENFHL6ORQQBR5YDUOU2QAY7OLQ5KYXOK2DW5DRGA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.036942675 (376h31m10.80758403s) [noise=true , address=104.36.82.9:4007 , piece_id=TMZJKWZDZHE4ANML45KWGRPDG5UOTNIYUZFDH3LJYZHMU6LBNTIA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.037006067 (376h31m10.807647412s) [noise=true , address=45.91.116.10:4082 , piece_id=DNZJWYESTZZY2QVM2SRD32AS6GDWSSO7ODNWMHBXJAGRNQ3FMSMQ, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.037346698 (376h31m10.807988043s) [noise=true , address=45.91.116.6:4975 , piece_id=LDM5B2S4NP43AXC3DJXTUM66Z6EM3SBKPP6JHTMLA7RZATZQ4QQA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.037776601 (376h31m10.808417936s) [noise=true , address=104.36.82.11:4114 , piece_id=5ISRL5WJST2SCUOPBB7KX53T3PKO2CH5X7QKCPGSKSDO3KA5BYHQ, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.038177858 (376h31m10.808819193s) [noise=true , address=104.36.82.14:4222 , piece_id=FU4BHM26LXD5XUTT333YO7ETORBXWP6QWCLUUJI2MBMPWLEFMGAA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.038258122 (376h31m10.808899467s) [noise=true , address=77.90.31.12:4592 , piece_id=J7LR65VEHL5D7XBO2HMEMFZVFBPEES5IL7FQTV5IJK4ZHEHGDKOA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.038562023 (376h31m10.809203359s) [noise=true , address=45.91.116.13:4153 , piece_id=UMTA7DVZBXEYRQQXBO33BCQL4W5L5RKJAYACZHYJMHCX6XORWSWQ, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.038582503 (376h31m10.809223838s) [noise=true , address=104.36.82.2:4730 , piece_id=EVW56CTEQD7BPUSYS2U5NXO4VRG6YZVEITV6OU5XKT3AU3MRWRYQ, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.038924036 (376h31m10.809565371s) [noise=true , address=104.36.82.8:4969 , piece_id=6IKLPKOPP2HBAOC5FAI5IE5SNW2B2UWNWT3Q6S4WWKGDLRCARVWA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.039254127 (376h31m10.809895453s) [noise=true , address=77.90.31.21:4878 , piece_id=4YNDQENFOKMHFJFVWWBC7UK4TQP745OUW7S2P43BICF2SNFAWIHA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.039603647 (376h31m10.810244982s) [noise=true , address=45.91.116.12:4227 , piece_id=23YQLIAPNPNMSA3LL67PNYCRDMWJ4GXZY3VF354XXUH5YEXFWVUA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:37.680432693 (376h31m12.451074028s) [upload=301225]: Next returned nil segment
Jul 9 01:41:37.680461809 (376h31m12.451103144s) [upload=301225]: Waiting for error group managing segments...
Jul 9 01:41:38.011139865 (376h31m12.7817812s) [noise=true , address=104.36.82.16:4294 , piece_id=CE4MNKGLLBYQAQ7KGVTTGUMWRTD2NTWF55NOPILQC5YPXDQT3Z6A, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.014224286 (376h31m12.784865621s) [noise=true , address=104.36.82.9:4007 , piece_id=TMZJKWZDZHE4ANML45KWGRPDG5UOTNIYUZFDH3LJYZHMU6LBNTIA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.116297430 (376h31m12.886938765s) [noise=true , address=45.91.116.6:4975 , piece_id=LDM5B2S4NP43AXC3DJXTUM66Z6EM3SBKPP6JHTMLA7RZATZQ4QQA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.118333195 (376h31m12.88897453s) [noise=true , address=45.91.116.13:4153 , piece_id=UMTA7DVZBXEYRQQXBO33BCQL4W5L5RKJAYACZHYJMHCX6XORWSWQ, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.130186111 (376h31m12.900827446s) [noise=true , address=45.91.116.8:4045 , piece_id=Z37SMFGUPVJVP6RNGUIDH6Q456OJFVZ7SNFFYT3NT3AOXVYR5TXA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.130610163 (376h31m12.901251498s) [noise=true , address=45.91.116.10:4082 , piece_id=DNZJWYESTZZY2QVM2SRD32AS6GDWSSO7ODNWMHBXJAGRNQ3FMSMQ, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.171163518 (376h31m12.941804854s) [noise=true , address=77.90.31.19:4843 , piece_id=JH6TQLWG4LCPENFHL6ORQQBR5YDUOU2QAY7OLQ5KYXOK2DW5DRGA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.187728911 (376h31m12.958370246s) [noise=true , address=77.90.31.12:4592 , piece_id=J7LR65VEHL5D7XBO2HMEMFZVFBPEES5IL7FQTV5IJK4ZHEHGDKOA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.190933191 (376h31m12.961574526s) [noise=true , address=77.90.31.21:4878 , piece_id=4YNDQENFOKMHFJFVWWBC7UK4TQP745OUW7S2P43BICF2SNFAWIHA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.218675778 (376h31m12.989317113s) [noise=true , address=45.91.116.12:4227 , piece_id=23YQLIAPNPNMSA3LL67PNYCRDMWJ4GXZY3VF354XXUH5YEXFWVUA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.225536314 (376h31m12.996177649s) [noise=true , address=45.91.116.14:4190 , piece_id=CZGKNHBIJP7HZIC65I6ESRFRQLBKTEAQJVVUQHHZHFO7TNNROJ4A, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:42:05.141496679 (376h31m39.912138074s) [noise=true , address=104.36.82.8:4969 , piece_id=6IKLPKOPP2HBAOC5FAI5IE5SNW2B2UWNWT3Q6S4WWKGDLRCARVWA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 01:42:05.141543177 (376h31m39.912184512s) [noise=true , address=104.36.82.14:4222 , piece_id=FU4BHM26LXD5XUTT333YO7ETORBXWP6QWCLUUJI2MBMPWLEFMGAA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 188.166.225.170:60278->104.36.82.14:4222: use of closed network connection; write tcp 188.166.225.170:60278->104.36.82.14:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60278->104.36.82.14:4222: use of closed network connection
Jul 9 01:42:05.141602951 (376h31m39.912244286s) [noise=true , address=104.36.82.5:4840 , piece_id=QKGEMZVR5P3B6ARIGS4VQDSWGAOVC5DU65Y3VLXUY2MAYVKDUQCQ, seg_pos={0 0}, upload=301225]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: write tcp 188.166.225.170:35250->104.36.82.5:4840: use of closed network connection; write tcp 188.166.225.170:35250->104.36.82.5:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35250->104.36.82.5:4840: use of closed network connection
Jul 9 01:42:05.141829285 (376h31m39.91247062s) [noise=true , address=104.36.82.11:4114 , piece_id=5ISRL5WJST2SCUOPBB7KX53T3PKO2CH5X7QKCPGSKSDO3KA5BYHQ, seg_pos={0 0}, upload=301225]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 188.166.225.170:54484->104.36.82.11:4114: use of closed network connection; write tcp 188.166.225.170:54484->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54484->104.36.82.11:4114: use of closed network connection
Jul 9 01:42:05.141852700 (376h31m39.912494025s) [noise=true , address=104.36.82.2:4730 , piece_id=EVW56CTEQD7BPUSYS2U5NXO4VRG6YZVEITV6OU5XKT3AU3MRWRYQ, seg_pos={0 0}, upload=301225]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 188.166.225.170:59748->104.36.82.2:4730: use of closed network connection; write tcp 188.166.225.170:59748->104.36.82.2:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59748->104.36.82.2:4730: use of closed network connection
Jul 9 01:42:05.141879792 (376h31m39.912521127s) [seg_pos={0 0}, upload=301225]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 9 01:42:05.366924712 (376h31m40.137566037s) [upload=301225]: Done uploading segments...
Jul 9 01:42:05.366949890 (376h31m40.137591225s) [upload=301225]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 5.938407595s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [.........X..........................X......X..........X.....] duration:
23.456191086s
FAIL (2025-07-09 01:48:53 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
s) [upload=301248]: Got next segment. Inline: false
Jul 9 01:43:24.322645491 (376h32m59.093286826s) [upload=301248]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 01:43:24.322650441 (376h32m59.093291777s) [upload=301248]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 01:43:24.769774209 (376h32m59.540415544s) [seg_pos={0 0}, upload=301248]: Begin upload segment...
Jul 9 01:43:24.769830437 (376h32m59.540471772s) [seg_pos={0 0}, upload=301248]: Done begin upload segment.
Jul 9 01:43:24.769852048 (376h32m59.540493373s) [noise=true , address=79.127.205.234:4614 , piece_id=JPPMN7KOKXFYF2YDK4RC7AU6KYV2UHQELFEG6ZZ6W5HQ2MKFVN5A, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.769864432 (376h32m59.540505767s) [noise=true , address=45.91.116.11:4123 , piece_id=WSMWN6I2QVBTO6OCRRDCMT3G3TESW3H2ZET5YNWFUPVENPN3UWKQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.770250630 (376h32m59.540891965s) [noise=true , address=104.36.82.22:4511 , piece_id=3AIKVWFRA4CLMPRDYFNX6EA7XO22MB7RQEOVNXPUVUCNFAHPSFCQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.770361543 (376h32m59.541002878s) [noise=true , address=77.90.31.21:4886 , piece_id=YRDZYEV6KU5YISF4DIS3GTZVBFRCZEOIW2DG7VCKHO26NRXMHMVQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.770542389 (376h32m59.541183724s) [noise=true , address=77.90.31.19:4848 , piece_id=RV7TBRZW2YH64X5CLXFHFTJD2WQCEM3P7OW3NYZ2IWC2E3DJDG5A, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.770626239 (376h32m59.541267594s) [noise=true , address=104.36.82.19:4411 , piece_id=R27NIJUXHNTQWK4IXXSVS2QC535JAZJ3K2WV4VANVOGKRYV7YADA, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.770818237 (376h32m59.541459572s) [noise=true , address=104.36.82.40:4955 , piece_id=3TVDS5QR5267W6VKWNXOPW6ZSTXISGXFBIPKK7E3S6ML7C4X4MUQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.770918087 (376h32m59.541559422s) [noise=true , address=104.36.82.12:4160 , piece_id=2Z3FQL5GTT26HC2GPKO5YPOLPB2Z75K4UG3FDPMOEDCCD6Q6MULQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.771127618 (376h32m59.541768953s) [noise=true , address=77.90.31.14:4677 , piece_id=Z7SGXIO722E3U7PWA4AWOUD5T27VSQUCG3ZBVP22WJRFSRKOEXAQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.771146836 (376h32m59.541788171s) [noise=true , address=77.90.31.12:4622 , piece_id=25K7AOPYFLZ6EDAA3SPHHE6S635PN5VB3ZK4LIULBHKQGYYQ3C7A, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.771367858 (376h32m59.542009193s) [noise=true , address=77.90.31.18:4825 , piece_id=NFOYMUTMKMKTPCAEPBTWL43ISOETOAM4WKVAUNHCETNW75IQLLMQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.771421060 (376h32m59.542062395s) [noise=true , address=77.90.31.14:4691 , piece_id=ELMXSLQT6TNOMGNUGM4BEHZ2P7OZWNSX6EOCHDAYOD7OIANFUVHA, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.771586527 (376h32m59.542227852s) [noise=true , address=64.49.13.169:4706 , piece_id=2CKVJCKZJH4F3VC3X7XJOWMYHVHU52POGAA7VMAU4YM6HTVHC5PA, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.771699443 (376h32m59.542340778s) [noise=true , address=104.36.82.44:4898 , piece_id=X47T6NYNODFSVKHJGWIH2WJT7BA3UVP7BEBYVKPPQBMN4QOQCJ6Q, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.771818129 (376h32m59.542459465s) [noise=true , address=104.36.82.34:4551 , piece_id=HWYKZWYEKMJT3JHDPHRWIAFKP5NZQ72OGR4SPR2I2SRRABLA663A, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.772055795 (376h32m59.54269713s) [noise=true , address=77.90.31.10:4543 , piece_id=QCKPGFCSQ3N57BWXMOSW5LAVA4KOI3EVUFU2OMPN7CMAOYAH4RWQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.772089298 (376h32m59.542730623s) [noise=true , address=104.36.82.23:4459 , piece_id=SOG4BGCMVK36OVMKHGIYGSVGX2ZZ2PWZV74S2LQS6IZQRGDYJVYA, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.772274673 (376h32m59.542916008s) [noise=true , address=77.90.31.15:4715 , piece_id=ITPPGCX32ZLU72XXE7RG6AYE3PM7IILVNPXM5A4NJ2S7P6U4Y2FA, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:25.540057386 (376h33m0.310698721s) [noise=true , address=64.49.13.169:4706 , piece_id=2CKVJCKZJH4F3VC3X7XJOWMYHVHU52POGAA7VMAU4YM6HTVHC5PA, seg_pos={0 0}, upload=301248]: Done uploading piece. err: ecclient: upload failed (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7, address:64.49.13.169:4706): protocol: storage node overloaded, request limit: 2000
Jul 9 01:43:25.878055136 (376h33m0.648696481s) [upload=301248]: Next returned nil segment
Jul 9 01:43:25.878073130 (376h33m0.648714465s) [upload=301248]: Waiting for error group managing segments...
Jul 9 01:43:26.175673001 (376h33m0.946314336s) [noise=true , address=79.127.205.234:4614 , piece_id=JPPMN7KOKXFYF2YDK4RC7AU6KYV2UHQELFEG6ZZ6W5HQ2MKFVN5A, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.774276626 (376h33m1.544917961s) [noise=true , address=77.90.31.12:4622 , piece_id=25K7AOPYFLZ6EDAA3SPHHE6S635PN5VB3ZK4LIULBHKQGYYQ3C7A, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.778019557 (376h33m1.548660892s) [noise=true , address=77.90.31.21:4886 , piece_id=YRDZYEV6KU5YISF4DIS3GTZVBFRCZEOIW2DG7VCKHO26NRXMHMVQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.779099603 (376h33m1.549740938s) [noise=true , address=77.90.31.14:4691 , piece_id=ELMXSLQT6TNOMGNUGM4BEHZ2P7OZWNSX6EOCHDAYOD7OIANFUVHA, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.779161091 (376h33m1.549802426s) [noise=true , address=77.90.31.18:4825 , piece_id=NFOYMUTMKMKTPCAEPBTWL43ISOETOAM4WKVAUNHCETNW75IQLLMQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.780045593 (376h33m1.550686928s) [noise=true , address=77.90.31.14:4677 , piece_id=Z7SGXIO722E3U7PWA4AWOUD5T27VSQUCG3ZBVP22WJRFSRKOEXAQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.795740620 (376h33m1.566381976s) [noise=true , address=77.90.31.10:4543 , piece_id=QCKPGFCSQ3N57BWXMOSW5LAVA4KOI3EVUFU2OMPN7CMAOYAH4RWQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.795758073 (376h33m1.566399408s) [noise=true , address=77.90.31.15:4715 , piece_id=ITPPGCX32ZLU72XXE7RG6AYE3PM7IILVNPXM5A4NJ2S7P6U4Y2FA, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.812585147 (376h33m1.583226483s) [noise=true , address=77.90.31.19:4848 , piece_id=RV7TBRZW2YH64X5CLXFHFTJD2WQCEM3P7OW3NYZ2IWC2E3DJDG5A, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:27.386630478 (376h33m2.157271803s) [noise=true , address=45.91.116.11:4123 , piece_id=WSMWN6I2QVBTO6OCRRDCMT3G3TESW3H2ZET5YNWFUPVENPN3UWKQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:37.603914421 (376h33m12.374555756s) [noise=true , address=104.36.82.34:4551 , piece_id=HWYKZWYEKMJT3JHDPHRWIAFKP5NZQ72OGR4SPR2I2SRRABLA663A, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:41.629246242 (376h33m16.399887567s) [noise=true , address=104.36.82.12:4160 , piece_id=2Z3FQL5GTT26HC2GPKO5YPOLPB2Z75K4UG3FDPMOEDCCD6Q6MULQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:44:10.222815170 (376h33m44.993456505s) [noise=true , address=104.36.82.22:4511 , piece_id=3AIKVWFRA4CLMPRDYFNX6EA7XO22MB7RQEOVNXPUVUCNFAHPSFCQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:44:21.573027994 (376h33m56.343669329s) [noise=true , address=104.36.82.23:4459 , piece_id=SOG4BGCMVK36OVMKHGIYGSVGX2ZZ2PWZV74S2LQS6IZQRGDYJVYA, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:44:21.573054295 (376h33m56.34369563s) [seg_pos={0 0}, upload=301248]: Segment reached optimal threshold of 14 pieces.
Jul 9 01:44:21.573140269 (376h33m56.343781604s) [noise=true , address=104.36.82.40:4955 , piece_id=3TVDS5QR5267W6VKWNXOPW6ZSTXISGXFBIPKK7E3S6ML7C4X4MUQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ssLq5bo7og4z8ZPd28f2H6WUExsWohBBZUnciGaH5KKpcQ25d): protocol: expected piece hash; context canceled; EOF
Jul 9 01:44:21.573153795 (376h33m56.34379513s) [noise=true , address=104.36.82.19:4411 , piece_id=R27NIJUXHNTQWK4IXXSVS2QC535JAZJ3K2WV4VANVOGKRYV7YADA, seg_pos={0 0}, upload=301248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nradx4SMYLkuJw3Ad6o9Sq1iNtQR6GDu8qg7tCzZzNcTjVfyd): protocol: expected piece hash; context canceled; EOF
Jul 9 01:44:21.573259487 (376h33m56.343900822s) [noise=true , address=104.36.82.44:4898 , piece_id=X47T6NYNODFSVKHJGWIH2WJT7BA3UVP7BEBYVKPPQBMN4QOQCJ6Q, seg_pos={0 0}, upload=301248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc): protocol: write tcp 188.166.225.170:40614->104.36.82.44:4898: use of closed network connection; write tcp 188.166.225.170:40614->104.36.82.44:4898: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40614->104.36.82.44:4898: use of closed network connection
Jul 9 01:44:21.573276430 (376h33m56.343917755s) [seg_pos={0 0}, upload=301248]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 01:44:21.573349429 (376h33m56.343990764s) [upload=301248]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 01:44:21.573353537 (376h33m56.343994873s) [upload=301248]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 01:44:21.865577192 (376h33m56.636218527s) [upload=301248]: Done uploading segments...
Jul 9 01:44:21.865594906 (376h33m56.636236232s) [upload=301248]: Upload finished. err: <nil>
Duration: 23.456206815s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
13 out of 60 is failed (threshold=3) [......................X......XX.XX..X......XXX....XX.....XX.] duration:
41.001559137s
FAIL (2025-07-09 01:48:53 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 41.001632136s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [.XX....X...X.X....X...X.X..X...X..X............X......X..X..] duration:
10.966750968s
FAIL (2025-07-09 01:48:50 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 10.966770384s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..X........................X......................X.........] duration:
5.640933546s
FAIL (2025-07-09 01:48:43 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Jul 9 01:41:35.784016383 (376h31m10.554657718s) [upload=301225]: Starting upload
Jul 9 01:41:35.784073732 (376h31m10.554715067s) [upload=301225]: Done starting upload
Jul 9 01:41:35.784180246 (376h31m10.554821571s) [upload=301225]: Uploading segments...
Jul 9 01:41:35.784275688 (376h31m10.554917023s) [upload=301225]: Got next segment. Inline: false
Jul 9 01:41:35.784309523 (376h31m10.554950848s) [upload=301225]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 01:41:35.784314523 (376h31m10.554955858s) [upload=301225]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 01:41:36.035706020 (376h31m10.806347375s) [seg_pos={0 0}, upload=301225]: Begin upload segment...
Jul 9 01:41:36.035755273 (376h31m10.806396598s) [seg_pos={0 0}, upload=301225]: Done begin upload segment.
Jul 9 01:41:36.035768209 (376h31m10.806409574s) [noise=true , address=45.91.116.14:4190 , piece_id=CZGKNHBIJP7HZIC65I6ESRFRQLBKTEAQJVVUQHHZHFO7TNNROJ4A, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.035782255 (376h31m10.80642359s) [noise=true , address=104.36.82.16:4294 , piece_id=CE4MNKGLLBYQAQ7KGVTTGUMWRTD2NTWF55NOPILQC5YPXDQT3Z6A, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.035859012 (376h31m10.806500367s) [noise=true , address=104.36.82.5:4840 , piece_id=QKGEMZVR5P3B6ARIGS4VQDSWGAOVC5DU65Y3VLXUY2MAYVKDUQCQ, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.036404877 (376h31m10.807046232s) [noise=true , address=45.91.116.8:4045 , piece_id=Z37SMFGUPVJVP6RNGUIDH6Q456OJFVZ7SNFFYT3NT3AOXVYR5TXA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.036420547 (376h31m10.807061892s) [noise=true , address=77.90.31.19:4843 , piece_id=JH6TQLWG4LCPENFHL6ORQQBR5YDUOU2QAY7OLQ5KYXOK2DW5DRGA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.036942675 (376h31m10.80758403s) [noise=true , address=104.36.82.9:4007 , piece_id=TMZJKWZDZHE4ANML45KWGRPDG5UOTNIYUZFDH3LJYZHMU6LBNTIA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.037006067 (376h31m10.807647412s) [noise=true , address=45.91.116.10:4082 , piece_id=DNZJWYESTZZY2QVM2SRD32AS6GDWSSO7ODNWMHBXJAGRNQ3FMSMQ, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.037346698 (376h31m10.807988043s) [noise=true , address=45.91.116.6:4975 , piece_id=LDM5B2S4NP43AXC3DJXTUM66Z6EM3SBKPP6JHTMLA7RZATZQ4QQA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.037776601 (376h31m10.808417936s) [noise=true , address=104.36.82.11:4114 , piece_id=5ISRL5WJST2SCUOPBB7KX53T3PKO2CH5X7QKCPGSKSDO3KA5BYHQ, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.038177858 (376h31m10.808819193s) [noise=true , address=104.36.82.14:4222 , piece_id=FU4BHM26LXD5XUTT333YO7ETORBXWP6QWCLUUJI2MBMPWLEFMGAA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.038258122 (376h31m10.808899467s) [noise=true , address=77.90.31.12:4592 , piece_id=J7LR65VEHL5D7XBO2HMEMFZVFBPEES5IL7FQTV5IJK4ZHEHGDKOA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.038562023 (376h31m10.809203359s) [noise=true , address=45.91.116.13:4153 , piece_id=UMTA7DVZBXEYRQQXBO33BCQL4W5L5RKJAYACZHYJMHCX6XORWSWQ, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.038582503 (376h31m10.809223838s) [noise=true , address=104.36.82.2:4730 , piece_id=EVW56CTEQD7BPUSYS2U5NXO4VRG6YZVEITV6OU5XKT3AU3MRWRYQ, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.038924036 (376h31m10.809565371s) [noise=true , address=104.36.82.8:4969 , piece_id=6IKLPKOPP2HBAOC5FAI5IE5SNW2B2UWNWT3Q6S4WWKGDLRCARVWA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.039254127 (376h31m10.809895453s) [noise=true , address=77.90.31.21:4878 , piece_id=4YNDQENFOKMHFJFVWWBC7UK4TQP745OUW7S2P43BICF2SNFAWIHA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.039603647 (376h31m10.810244982s) [noise=true , address=45.91.116.12:4227 , piece_id=23YQLIAPNPNMSA3LL67PNYCRDMWJ4GXZY3VF354XXUH5YEXFWVUA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:37.680432693 (376h31m12.451074028s) [upload=301225]: Next returned nil segment
Jul 9 01:41:37.680461809 (376h31m12.451103144s) [upload=301225]: Waiting for error group managing segments...
Jul 9 01:41:38.011139865 (376h31m12.7817812s) [noise=true , address=104.36.82.16:4294 , piece_id=CE4MNKGLLBYQAQ7KGVTTGUMWRTD2NTWF55NOPILQC5YPXDQT3Z6A, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.014224286 (376h31m12.784865621s) [noise=true , address=104.36.82.9:4007 , piece_id=TMZJKWZDZHE4ANML45KWGRPDG5UOTNIYUZFDH3LJYZHMU6LBNTIA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.116297430 (376h31m12.886938765s) [noise=true , address=45.91.116.6:4975 , piece_id=LDM5B2S4NP43AXC3DJXTUM66Z6EM3SBKPP6JHTMLA7RZATZQ4QQA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.118333195 (376h31m12.88897453s) [noise=true , address=45.91.116.13:4153 , piece_id=UMTA7DVZBXEYRQQXBO33BCQL4W5L5RKJAYACZHYJMHCX6XORWSWQ, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.130186111 (376h31m12.900827446s) [noise=true , address=45.91.116.8:4045 , piece_id=Z37SMFGUPVJVP6RNGUIDH6Q456OJFVZ7SNFFYT3NT3AOXVYR5TXA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.130610163 (376h31m12.901251498s) [noise=true , address=45.91.116.10:4082 , piece_id=DNZJWYESTZZY2QVM2SRD32AS6GDWSSO7ODNWMHBXJAGRNQ3FMSMQ, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.171163518 (376h31m12.941804854s) [noise=true , address=77.90.31.19:4843 , piece_id=JH6TQLWG4LCPENFHL6ORQQBR5YDUOU2QAY7OLQ5KYXOK2DW5DRGA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.187728911 (376h31m12.958370246s) [noise=true , address=77.90.31.12:4592 , piece_id=J7LR65VEHL5D7XBO2HMEMFZVFBPEES5IL7FQTV5IJK4ZHEHGDKOA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.190933191 (376h31m12.961574526s) [noise=true , address=77.90.31.21:4878 , piece_id=4YNDQENFOKMHFJFVWWBC7UK4TQP745OUW7S2P43BICF2SNFAWIHA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.218675778 (376h31m12.989317113s) [noise=true , address=45.91.116.12:4227 , piece_id=23YQLIAPNPNMSA3LL67PNYCRDMWJ4GXZY3VF354XXUH5YEXFWVUA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.225536314 (376h31m12.996177649s) [noise=true , address=45.91.116.14:4190 , piece_id=CZGKNHBIJP7HZIC65I6ESRFRQLBKTEAQJVVUQHHZHFO7TNNROJ4A, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:42:05.141496679 (376h31m39.912138074s) [noise=true , address=104.36.82.8:4969 , piece_id=6IKLPKOPP2HBAOC5FAI5IE5SNW2B2UWNWT3Q6S4WWKGDLRCARVWA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 01:42:05.141543177 (376h31m39.912184512s) [noise=true , address=104.36.82.14:4222 , piece_id=FU4BHM26LXD5XUTT333YO7ETORBXWP6QWCLUUJI2MBMPWLEFMGAA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 188.166.225.170:60278->104.36.82.14:4222: use of closed network connection; write tcp 188.166.225.170:60278->104.36.82.14:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60278->104.36.82.14:4222: use of closed network connection
Jul 9 01:42:05.141602951 (376h31m39.912244286s) [noise=true , address=104.36.82.5:4840 , piece_id=QKGEMZVR5P3B6ARIGS4VQDSWGAOVC5DU65Y3VLXUY2MAYVKDUQCQ, seg_pos={0 0}, upload=301225]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: write tcp 188.166.225.170:35250->104.36.82.5:4840: use of closed network connection; write tcp 188.166.225.170:35250->104.36.82.5:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35250->104.36.82.5:4840: use of closed network connection
Jul 9 01:42:05.141829285 (376h31m39.91247062s) [noise=true , address=104.36.82.11:4114 , piece_id=5ISRL5WJST2SCUOPBB7KX53T3PKO2CH5X7QKCPGSKSDO3KA5BYHQ, seg_pos={0 0}, upload=301225]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 188.166.225.170:54484->104.36.82.11:4114: use of closed network connection; write tcp 188.166.225.170:54484->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54484->104.36.82.11:4114: use of closed network connection
Jul 9 01:42:05.141852700 (376h31m39.912494025s) [noise=true , address=104.36.82.2:4730 , piece_id=EVW56CTEQD7BPUSYS2U5NXO4VRG6YZVEITV6OU5XKT3AU3MRWRYQ, seg_pos={0 0}, upload=301225]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 188.166.225.170:59748->104.36.82.2:4730: use of closed network connection; write tcp 188.166.225.170:59748->104.36.82.2:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59748->104.36.82.2:4730: use of closed network connection
Jul 9 01:42:05.141879792 (376h31m39.912521127s) [seg_pos={0 0}, upload=301225]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 9 01:42:05.366924712 (376h31m40.137566037s) [upload=301225]: Done uploading segments...
Jul 9 01:42:05.366949890 (376h31m40.137591225s) [upload=301225]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 5.640960767s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [..........X..........................X......X..........X....] duration:
58.540004333s
FAIL (2025-07-09 01:48:29 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
s) [upload=301248]: Got next segment. Inline: false
Jul 9 01:43:24.322645491 (376h32m59.093286826s) [upload=301248]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 01:43:24.322650441 (376h32m59.093291777s) [upload=301248]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 01:43:24.769774209 (376h32m59.540415544s) [seg_pos={0 0}, upload=301248]: Begin upload segment...
Jul 9 01:43:24.769830437 (376h32m59.540471772s) [seg_pos={0 0}, upload=301248]: Done begin upload segment.
Jul 9 01:43:24.769852048 (376h32m59.540493373s) [noise=true , address=79.127.205.234:4614 , piece_id=JPPMN7KOKXFYF2YDK4RC7AU6KYV2UHQELFEG6ZZ6W5HQ2MKFVN5A, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.769864432 (376h32m59.540505767s) [noise=true , address=45.91.116.11:4123 , piece_id=WSMWN6I2QVBTO6OCRRDCMT3G3TESW3H2ZET5YNWFUPVENPN3UWKQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.770250630 (376h32m59.540891965s) [noise=true , address=104.36.82.22:4511 , piece_id=3AIKVWFRA4CLMPRDYFNX6EA7XO22MB7RQEOVNXPUVUCNFAHPSFCQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.770361543 (376h32m59.541002878s) [noise=true , address=77.90.31.21:4886 , piece_id=YRDZYEV6KU5YISF4DIS3GTZVBFRCZEOIW2DG7VCKHO26NRXMHMVQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.770542389 (376h32m59.541183724s) [noise=true , address=77.90.31.19:4848 , piece_id=RV7TBRZW2YH64X5CLXFHFTJD2WQCEM3P7OW3NYZ2IWC2E3DJDG5A, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.770626239 (376h32m59.541267594s) [noise=true , address=104.36.82.19:4411 , piece_id=R27NIJUXHNTQWK4IXXSVS2QC535JAZJ3K2WV4VANVOGKRYV7YADA, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.770818237 (376h32m59.541459572s) [noise=true , address=104.36.82.40:4955 , piece_id=3TVDS5QR5267W6VKWNXOPW6ZSTXISGXFBIPKK7E3S6ML7C4X4MUQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.770918087 (376h32m59.541559422s) [noise=true , address=104.36.82.12:4160 , piece_id=2Z3FQL5GTT26HC2GPKO5YPOLPB2Z75K4UG3FDPMOEDCCD6Q6MULQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.771127618 (376h32m59.541768953s) [noise=true , address=77.90.31.14:4677 , piece_id=Z7SGXIO722E3U7PWA4AWOUD5T27VSQUCG3ZBVP22WJRFSRKOEXAQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.771146836 (376h32m59.541788171s) [noise=true , address=77.90.31.12:4622 , piece_id=25K7AOPYFLZ6EDAA3SPHHE6S635PN5VB3ZK4LIULBHKQGYYQ3C7A, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.771367858 (376h32m59.542009193s) [noise=true , address=77.90.31.18:4825 , piece_id=NFOYMUTMKMKTPCAEPBTWL43ISOETOAM4WKVAUNHCETNW75IQLLMQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.771421060 (376h32m59.542062395s) [noise=true , address=77.90.31.14:4691 , piece_id=ELMXSLQT6TNOMGNUGM4BEHZ2P7OZWNSX6EOCHDAYOD7OIANFUVHA, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.771586527 (376h32m59.542227852s) [noise=true , address=64.49.13.169:4706 , piece_id=2CKVJCKZJH4F3VC3X7XJOWMYHVHU52POGAA7VMAU4YM6HTVHC5PA, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.771699443 (376h32m59.542340778s) [noise=true , address=104.36.82.44:4898 , piece_id=X47T6NYNODFSVKHJGWIH2WJT7BA3UVP7BEBYVKPPQBMN4QOQCJ6Q, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.771818129 (376h32m59.542459465s) [noise=true , address=104.36.82.34:4551 , piece_id=HWYKZWYEKMJT3JHDPHRWIAFKP5NZQ72OGR4SPR2I2SRRABLA663A, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.772055795 (376h32m59.54269713s) [noise=true , address=77.90.31.10:4543 , piece_id=QCKPGFCSQ3N57BWXMOSW5LAVA4KOI3EVUFU2OMPN7CMAOYAH4RWQ, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.772089298 (376h32m59.542730623s) [noise=true , address=104.36.82.23:4459 , piece_id=SOG4BGCMVK36OVMKHGIYGSVGX2ZZ2PWZV74S2LQS6IZQRGDYJVYA, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:24.772274673 (376h32m59.542916008s) [noise=true , address=77.90.31.15:4715 , piece_id=ITPPGCX32ZLU72XXE7RG6AYE3PM7IILVNPXM5A4NJ2S7P6U4Y2FA, seg_pos={0 0}, upload=301248]: Uploading piece...
Jul 9 01:43:25.540057386 (376h33m0.310698721s) [noise=true , address=64.49.13.169:4706 , piece_id=2CKVJCKZJH4F3VC3X7XJOWMYHVHU52POGAA7VMAU4YM6HTVHC5PA, seg_pos={0 0}, upload=301248]: Done uploading piece. err: ecclient: upload failed (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7, address:64.49.13.169:4706): protocol: storage node overloaded, request limit: 2000
Jul 9 01:43:25.878055136 (376h33m0.648696481s) [upload=301248]: Next returned nil segment
Jul 9 01:43:25.878073130 (376h33m0.648714465s) [upload=301248]: Waiting for error group managing segments...
Jul 9 01:43:26.175673001 (376h33m0.946314336s) [noise=true , address=79.127.205.234:4614 , piece_id=JPPMN7KOKXFYF2YDK4RC7AU6KYV2UHQELFEG6ZZ6W5HQ2MKFVN5A, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.774276626 (376h33m1.544917961s) [noise=true , address=77.90.31.12:4622 , piece_id=25K7AOPYFLZ6EDAA3SPHHE6S635PN5VB3ZK4LIULBHKQGYYQ3C7A, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.778019557 (376h33m1.548660892s) [noise=true , address=77.90.31.21:4886 , piece_id=YRDZYEV6KU5YISF4DIS3GTZVBFRCZEOIW2DG7VCKHO26NRXMHMVQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.779099603 (376h33m1.549740938s) [noise=true , address=77.90.31.14:4691 , piece_id=ELMXSLQT6TNOMGNUGM4BEHZ2P7OZWNSX6EOCHDAYOD7OIANFUVHA, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.779161091 (376h33m1.549802426s) [noise=true , address=77.90.31.18:4825 , piece_id=NFOYMUTMKMKTPCAEPBTWL43ISOETOAM4WKVAUNHCETNW75IQLLMQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.780045593 (376h33m1.550686928s) [noise=true , address=77.90.31.14:4677 , piece_id=Z7SGXIO722E3U7PWA4AWOUD5T27VSQUCG3ZBVP22WJRFSRKOEXAQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.795740620 (376h33m1.566381976s) [noise=true , address=77.90.31.10:4543 , piece_id=QCKPGFCSQ3N57BWXMOSW5LAVA4KOI3EVUFU2OMPN7CMAOYAH4RWQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.795758073 (376h33m1.566399408s) [noise=true , address=77.90.31.15:4715 , piece_id=ITPPGCX32ZLU72XXE7RG6AYE3PM7IILVNPXM5A4NJ2S7P6U4Y2FA, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:26.812585147 (376h33m1.583226483s) [noise=true , address=77.90.31.19:4848 , piece_id=RV7TBRZW2YH64X5CLXFHFTJD2WQCEM3P7OW3NYZ2IWC2E3DJDG5A, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:27.386630478 (376h33m2.157271803s) [noise=true , address=45.91.116.11:4123 , piece_id=WSMWN6I2QVBTO6OCRRDCMT3G3TESW3H2ZET5YNWFUPVENPN3UWKQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:37.603914421 (376h33m12.374555756s) [noise=true , address=104.36.82.34:4551 , piece_id=HWYKZWYEKMJT3JHDPHRWIAFKP5NZQ72OGR4SPR2I2SRRABLA663A, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:43:41.629246242 (376h33m16.399887567s) [noise=true , address=104.36.82.12:4160 , piece_id=2Z3FQL5GTT26HC2GPKO5YPOLPB2Z75K4UG3FDPMOEDCCD6Q6MULQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:44:10.222815170 (376h33m44.993456505s) [noise=true , address=104.36.82.22:4511 , piece_id=3AIKVWFRA4CLMPRDYFNX6EA7XO22MB7RQEOVNXPUVUCNFAHPSFCQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:44:21.573027994 (376h33m56.343669329s) [noise=true , address=104.36.82.23:4459 , piece_id=SOG4BGCMVK36OVMKHGIYGSVGX2ZZ2PWZV74S2LQS6IZQRGDYJVYA, seg_pos={0 0}, upload=301248]: Done uploading piece. err: <nil>
Jul 9 01:44:21.573054295 (376h33m56.34369563s) [seg_pos={0 0}, upload=301248]: Segment reached optimal threshold of 14 pieces.
Jul 9 01:44:21.573140269 (376h33m56.343781604s) [noise=true , address=104.36.82.40:4955 , piece_id=3TVDS5QR5267W6VKWNXOPW6ZSTXISGXFBIPKK7E3S6ML7C4X4MUQ, seg_pos={0 0}, upload=301248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ssLq5bo7og4z8ZPd28f2H6WUExsWohBBZUnciGaH5KKpcQ25d): protocol: expected piece hash; context canceled; EOF
Jul 9 01:44:21.573153795 (376h33m56.34379513s) [noise=true , address=104.36.82.19:4411 , piece_id=R27NIJUXHNTQWK4IXXSVS2QC535JAZJ3K2WV4VANVOGKRYV7YADA, seg_pos={0 0}, upload=301248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nradx4SMYLkuJw3Ad6o9Sq1iNtQR6GDu8qg7tCzZzNcTjVfyd): protocol: expected piece hash; context canceled; EOF
Jul 9 01:44:21.573259487 (376h33m56.343900822s) [noise=true , address=104.36.82.44:4898 , piece_id=X47T6NYNODFSVKHJGWIH2WJT7BA3UVP7BEBYVKPPQBMN4QOQCJ6Q, seg_pos={0 0}, upload=301248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qizevYxbpyxfYKv5wXfWJwdHKXMLbPXfgDWxihHMjCGdpKPqc): protocol: write tcp 188.166.225.170:40614->104.36.82.44:4898: use of closed network connection; write tcp 188.166.225.170:40614->104.36.82.44:4898: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40614->104.36.82.44:4898: use of closed network connection
Jul 9 01:44:21.573276430 (376h33m56.343917755s) [seg_pos={0 0}, upload=301248]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 01:44:21.573349429 (376h33m56.343990764s) [upload=301248]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 01:44:21.573353537 (376h33m56.343994873s) [upload=301248]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 01:44:21.865577192 (376h33m56.636218527s) [upload=301248]: Done uploading segments...
Jul 9 01:44:21.865594906 (376h33m56.636236232s) [upload=301248]: Upload finished. err: <nil>
Duration: 58.540027327s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [...X........................X......................X........] duration:
5.715830164s
FAIL (2025-07-09 01:48:08 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Jul 9 01:41:35.784016383 (376h31m10.554657718s) [upload=301225]: Starting upload
Jul 9 01:41:35.784073732 (376h31m10.554715067s) [upload=301225]: Done starting upload
Jul 9 01:41:35.784180246 (376h31m10.554821571s) [upload=301225]: Uploading segments...
Jul 9 01:41:35.784275688 (376h31m10.554917023s) [upload=301225]: Got next segment. Inline: false
Jul 9 01:41:35.784309523 (376h31m10.554950848s) [upload=301225]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 01:41:35.784314523 (376h31m10.554955858s) [upload=301225]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 01:41:36.035706020 (376h31m10.806347375s) [seg_pos={0 0}, upload=301225]: Begin upload segment...
Jul 9 01:41:36.035755273 (376h31m10.806396598s) [seg_pos={0 0}, upload=301225]: Done begin upload segment.
Jul 9 01:41:36.035768209 (376h31m10.806409574s) [noise=true , address=45.91.116.14:4190 , piece_id=CZGKNHBIJP7HZIC65I6ESRFRQLBKTEAQJVVUQHHZHFO7TNNROJ4A, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.035782255 (376h31m10.80642359s) [noise=true , address=104.36.82.16:4294 , piece_id=CE4MNKGLLBYQAQ7KGVTTGUMWRTD2NTWF55NOPILQC5YPXDQT3Z6A, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.035859012 (376h31m10.806500367s) [noise=true , address=104.36.82.5:4840 , piece_id=QKGEMZVR5P3B6ARIGS4VQDSWGAOVC5DU65Y3VLXUY2MAYVKDUQCQ, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.036404877 (376h31m10.807046232s) [noise=true , address=45.91.116.8:4045 , piece_id=Z37SMFGUPVJVP6RNGUIDH6Q456OJFVZ7SNFFYT3NT3AOXVYR5TXA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.036420547 (376h31m10.807061892s) [noise=true , address=77.90.31.19:4843 , piece_id=JH6TQLWG4LCPENFHL6ORQQBR5YDUOU2QAY7OLQ5KYXOK2DW5DRGA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.036942675 (376h31m10.80758403s) [noise=true , address=104.36.82.9:4007 , piece_id=TMZJKWZDZHE4ANML45KWGRPDG5UOTNIYUZFDH3LJYZHMU6LBNTIA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.037006067 (376h31m10.807647412s) [noise=true , address=45.91.116.10:4082 , piece_id=DNZJWYESTZZY2QVM2SRD32AS6GDWSSO7ODNWMHBXJAGRNQ3FMSMQ, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.037346698 (376h31m10.807988043s) [noise=true , address=45.91.116.6:4975 , piece_id=LDM5B2S4NP43AXC3DJXTUM66Z6EM3SBKPP6JHTMLA7RZATZQ4QQA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.037776601 (376h31m10.808417936s) [noise=true , address=104.36.82.11:4114 , piece_id=5ISRL5WJST2SCUOPBB7KX53T3PKO2CH5X7QKCPGSKSDO3KA5BYHQ, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.038177858 (376h31m10.808819193s) [noise=true , address=104.36.82.14:4222 , piece_id=FU4BHM26LXD5XUTT333YO7ETORBXWP6QWCLUUJI2MBMPWLEFMGAA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.038258122 (376h31m10.808899467s) [noise=true , address=77.90.31.12:4592 , piece_id=J7LR65VEHL5D7XBO2HMEMFZVFBPEES5IL7FQTV5IJK4ZHEHGDKOA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.038562023 (376h31m10.809203359s) [noise=true , address=45.91.116.13:4153 , piece_id=UMTA7DVZBXEYRQQXBO33BCQL4W5L5RKJAYACZHYJMHCX6XORWSWQ, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.038582503 (376h31m10.809223838s) [noise=true , address=104.36.82.2:4730 , piece_id=EVW56CTEQD7BPUSYS2U5NXO4VRG6YZVEITV6OU5XKT3AU3MRWRYQ, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.038924036 (376h31m10.809565371s) [noise=true , address=104.36.82.8:4969 , piece_id=6IKLPKOPP2HBAOC5FAI5IE5SNW2B2UWNWT3Q6S4WWKGDLRCARVWA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.039254127 (376h31m10.809895453s) [noise=true , address=77.90.31.21:4878 , piece_id=4YNDQENFOKMHFJFVWWBC7UK4TQP745OUW7S2P43BICF2SNFAWIHA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.039603647 (376h31m10.810244982s) [noise=true , address=45.91.116.12:4227 , piece_id=23YQLIAPNPNMSA3LL67PNYCRDMWJ4GXZY3VF354XXUH5YEXFWVUA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:37.680432693 (376h31m12.451074028s) [upload=301225]: Next returned nil segment
Jul 9 01:41:37.680461809 (376h31m12.451103144s) [upload=301225]: Waiting for error group managing segments...
Jul 9 01:41:38.011139865 (376h31m12.7817812s) [noise=true , address=104.36.82.16:4294 , piece_id=CE4MNKGLLBYQAQ7KGVTTGUMWRTD2NTWF55NOPILQC5YPXDQT3Z6A, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.014224286 (376h31m12.784865621s) [noise=true , address=104.36.82.9:4007 , piece_id=TMZJKWZDZHE4ANML45KWGRPDG5UOTNIYUZFDH3LJYZHMU6LBNTIA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.116297430 (376h31m12.886938765s) [noise=true , address=45.91.116.6:4975 , piece_id=LDM5B2S4NP43AXC3DJXTUM66Z6EM3SBKPP6JHTMLA7RZATZQ4QQA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.118333195 (376h31m12.88897453s) [noise=true , address=45.91.116.13:4153 , piece_id=UMTA7DVZBXEYRQQXBO33BCQL4W5L5RKJAYACZHYJMHCX6XORWSWQ, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.130186111 (376h31m12.900827446s) [noise=true , address=45.91.116.8:4045 , piece_id=Z37SMFGUPVJVP6RNGUIDH6Q456OJFVZ7SNFFYT3NT3AOXVYR5TXA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.130610163 (376h31m12.901251498s) [noise=true , address=45.91.116.10:4082 , piece_id=DNZJWYESTZZY2QVM2SRD32AS6GDWSSO7ODNWMHBXJAGRNQ3FMSMQ, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.171163518 (376h31m12.941804854s) [noise=true , address=77.90.31.19:4843 , piece_id=JH6TQLWG4LCPENFHL6ORQQBR5YDUOU2QAY7OLQ5KYXOK2DW5DRGA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.187728911 (376h31m12.958370246s) [noise=true , address=77.90.31.12:4592 , piece_id=J7LR65VEHL5D7XBO2HMEMFZVFBPEES5IL7FQTV5IJK4ZHEHGDKOA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.190933191 (376h31m12.961574526s) [noise=true , address=77.90.31.21:4878 , piece_id=4YNDQENFOKMHFJFVWWBC7UK4TQP745OUW7S2P43BICF2SNFAWIHA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.218675778 (376h31m12.989317113s) [noise=true , address=45.91.116.12:4227 , piece_id=23YQLIAPNPNMSA3LL67PNYCRDMWJ4GXZY3VF354XXUH5YEXFWVUA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.225536314 (376h31m12.996177649s) [noise=true , address=45.91.116.14:4190 , piece_id=CZGKNHBIJP7HZIC65I6ESRFRQLBKTEAQJVVUQHHZHFO7TNNROJ4A, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:42:05.141496679 (376h31m39.912138074s) [noise=true , address=104.36.82.8:4969 , piece_id=6IKLPKOPP2HBAOC5FAI5IE5SNW2B2UWNWT3Q6S4WWKGDLRCARVWA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 01:42:05.141543177 (376h31m39.912184512s) [noise=true , address=104.36.82.14:4222 , piece_id=FU4BHM26LXD5XUTT333YO7ETORBXWP6QWCLUUJI2MBMPWLEFMGAA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 188.166.225.170:60278->104.36.82.14:4222: use of closed network connection; write tcp 188.166.225.170:60278->104.36.82.14:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60278->104.36.82.14:4222: use of closed network connection
Jul 9 01:42:05.141602951 (376h31m39.912244286s) [noise=true , address=104.36.82.5:4840 , piece_id=QKGEMZVR5P3B6ARIGS4VQDSWGAOVC5DU65Y3VLXUY2MAYVKDUQCQ, seg_pos={0 0}, upload=301225]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: write tcp 188.166.225.170:35250->104.36.82.5:4840: use of closed network connection; write tcp 188.166.225.170:35250->104.36.82.5:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35250->104.36.82.5:4840: use of closed network connection
Jul 9 01:42:05.141829285 (376h31m39.91247062s) [noise=true , address=104.36.82.11:4114 , piece_id=5ISRL5WJST2SCUOPBB7KX53T3PKO2CH5X7QKCPGSKSDO3KA5BYHQ, seg_pos={0 0}, upload=301225]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 188.166.225.170:54484->104.36.82.11:4114: use of closed network connection; write tcp 188.166.225.170:54484->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54484->104.36.82.11:4114: use of closed network connection
Jul 9 01:42:05.141852700 (376h31m39.912494025s) [noise=true , address=104.36.82.2:4730 , piece_id=EVW56CTEQD7BPUSYS2U5NXO4VRG6YZVEITV6OU5XKT3AU3MRWRYQ, seg_pos={0 0}, upload=301225]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 188.166.225.170:59748->104.36.82.2:4730: use of closed network connection; write tcp 188.166.225.170:59748->104.36.82.2:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59748->104.36.82.2:4730: use of closed network connection
Jul 9 01:42:05.141879792 (376h31m39.912521127s) [seg_pos={0 0}, upload=301225]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 9 01:42:05.366924712 (376h31m40.137566037s) [upload=301225]: Done uploading segments...
Jul 9 01:42:05.366949890 (376h31m40.137591225s) [upload=301225]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 5.715857747s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [..XX....X...X.X....X...X.X..X...X..X............X......X..X.] duration:
19.587445388s
FAIL (2025-07-09 01:48:05 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 19.587466619s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
13 out of 60 is failed (threshold=3) [.......................X......XX.XX..X......XXX....XX.....XX] duration:
1m0.001351977s
FAIL (2025-07-09 01:48:05 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.001392594s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [....X........................X......................X.......] duration:
5.65379602s
FAIL (2025-07-09 01:47:40 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Jul 9 01:41:35.784016383 (376h31m10.554657718s) [upload=301225]: Starting upload
Jul 9 01:41:35.784073732 (376h31m10.554715067s) [upload=301225]: Done starting upload
Jul 9 01:41:35.784180246 (376h31m10.554821571s) [upload=301225]: Uploading segments...
Jul 9 01:41:35.784275688 (376h31m10.554917023s) [upload=301225]: Got next segment. Inline: false
Jul 9 01:41:35.784309523 (376h31m10.554950848s) [upload=301225]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 01:41:35.784314523 (376h31m10.554955858s) [upload=301225]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 01:41:36.035706020 (376h31m10.806347375s) [seg_pos={0 0}, upload=301225]: Begin upload segment...
Jul 9 01:41:36.035755273 (376h31m10.806396598s) [seg_pos={0 0}, upload=301225]: Done begin upload segment.
Jul 9 01:41:36.035768209 (376h31m10.806409574s) [noise=true , address=45.91.116.14:4190 , piece_id=CZGKNHBIJP7HZIC65I6ESRFRQLBKTEAQJVVUQHHZHFO7TNNROJ4A, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.035782255 (376h31m10.80642359s) [noise=true , address=104.36.82.16:4294 , piece_id=CE4MNKGLLBYQAQ7KGVTTGUMWRTD2NTWF55NOPILQC5YPXDQT3Z6A, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.035859012 (376h31m10.806500367s) [noise=true , address=104.36.82.5:4840 , piece_id=QKGEMZVR5P3B6ARIGS4VQDSWGAOVC5DU65Y3VLXUY2MAYVKDUQCQ, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.036404877 (376h31m10.807046232s) [noise=true , address=45.91.116.8:4045 , piece_id=Z37SMFGUPVJVP6RNGUIDH6Q456OJFVZ7SNFFYT3NT3AOXVYR5TXA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.036420547 (376h31m10.807061892s) [noise=true , address=77.90.31.19:4843 , piece_id=JH6TQLWG4LCPENFHL6ORQQBR5YDUOU2QAY7OLQ5KYXOK2DW5DRGA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.036942675 (376h31m10.80758403s) [noise=true , address=104.36.82.9:4007 , piece_id=TMZJKWZDZHE4ANML45KWGRPDG5UOTNIYUZFDH3LJYZHMU6LBNTIA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.037006067 (376h31m10.807647412s) [noise=true , address=45.91.116.10:4082 , piece_id=DNZJWYESTZZY2QVM2SRD32AS6GDWSSO7ODNWMHBXJAGRNQ3FMSMQ, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.037346698 (376h31m10.807988043s) [noise=true , address=45.91.116.6:4975 , piece_id=LDM5B2S4NP43AXC3DJXTUM66Z6EM3SBKPP6JHTMLA7RZATZQ4QQA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.037776601 (376h31m10.808417936s) [noise=true , address=104.36.82.11:4114 , piece_id=5ISRL5WJST2SCUOPBB7KX53T3PKO2CH5X7QKCPGSKSDO3KA5BYHQ, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.038177858 (376h31m10.808819193s) [noise=true , address=104.36.82.14:4222 , piece_id=FU4BHM26LXD5XUTT333YO7ETORBXWP6QWCLUUJI2MBMPWLEFMGAA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.038258122 (376h31m10.808899467s) [noise=true , address=77.90.31.12:4592 , piece_id=J7LR65VEHL5D7XBO2HMEMFZVFBPEES5IL7FQTV5IJK4ZHEHGDKOA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.038562023 (376h31m10.809203359s) [noise=true , address=45.91.116.13:4153 , piece_id=UMTA7DVZBXEYRQQXBO33BCQL4W5L5RKJAYACZHYJMHCX6XORWSWQ, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.038582503 (376h31m10.809223838s) [noise=true , address=104.36.82.2:4730 , piece_id=EVW56CTEQD7BPUSYS2U5NXO4VRG6YZVEITV6OU5XKT3AU3MRWRYQ, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.038924036 (376h31m10.809565371s) [noise=true , address=104.36.82.8:4969 , piece_id=6IKLPKOPP2HBAOC5FAI5IE5SNW2B2UWNWT3Q6S4WWKGDLRCARVWA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.039254127 (376h31m10.809895453s) [noise=true , address=77.90.31.21:4878 , piece_id=4YNDQENFOKMHFJFVWWBC7UK4TQP745OUW7S2P43BICF2SNFAWIHA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:36.039603647 (376h31m10.810244982s) [noise=true , address=45.91.116.12:4227 , piece_id=23YQLIAPNPNMSA3LL67PNYCRDMWJ4GXZY3VF354XXUH5YEXFWVUA, seg_pos={0 0}, upload=301225]: Uploading piece...
Jul 9 01:41:37.680432693 (376h31m12.451074028s) [upload=301225]: Next returned nil segment
Jul 9 01:41:37.680461809 (376h31m12.451103144s) [upload=301225]: Waiting for error group managing segments...
Jul 9 01:41:38.011139865 (376h31m12.7817812s) [noise=true , address=104.36.82.16:4294 , piece_id=CE4MNKGLLBYQAQ7KGVTTGUMWRTD2NTWF55NOPILQC5YPXDQT3Z6A, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.014224286 (376h31m12.784865621s) [noise=true , address=104.36.82.9:4007 , piece_id=TMZJKWZDZHE4ANML45KWGRPDG5UOTNIYUZFDH3LJYZHMU6LBNTIA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.116297430 (376h31m12.886938765s) [noise=true , address=45.91.116.6:4975 , piece_id=LDM5B2S4NP43AXC3DJXTUM66Z6EM3SBKPP6JHTMLA7RZATZQ4QQA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.118333195 (376h31m12.88897453s) [noise=true , address=45.91.116.13:4153 , piece_id=UMTA7DVZBXEYRQQXBO33BCQL4W5L5RKJAYACZHYJMHCX6XORWSWQ, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.130186111 (376h31m12.900827446s) [noise=true , address=45.91.116.8:4045 , piece_id=Z37SMFGUPVJVP6RNGUIDH6Q456OJFVZ7SNFFYT3NT3AOXVYR5TXA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.130610163 (376h31m12.901251498s) [noise=true , address=45.91.116.10:4082 , piece_id=DNZJWYESTZZY2QVM2SRD32AS6GDWSSO7ODNWMHBXJAGRNQ3FMSMQ, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.171163518 (376h31m12.941804854s) [noise=true , address=77.90.31.19:4843 , piece_id=JH6TQLWG4LCPENFHL6ORQQBR5YDUOU2QAY7OLQ5KYXOK2DW5DRGA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.187728911 (376h31m12.958370246s) [noise=true , address=77.90.31.12:4592 , piece_id=J7LR65VEHL5D7XBO2HMEMFZVFBPEES5IL7FQTV5IJK4ZHEHGDKOA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.190933191 (376h31m12.961574526s) [noise=true , address=77.90.31.21:4878 , piece_id=4YNDQENFOKMHFJFVWWBC7UK4TQP745OUW7S2P43BICF2SNFAWIHA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.218675778 (376h31m12.989317113s) [noise=true , address=45.91.116.12:4227 , piece_id=23YQLIAPNPNMSA3LL67PNYCRDMWJ4GXZY3VF354XXUH5YEXFWVUA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:41:38.225536314 (376h31m12.996177649s) [noise=true , address=45.91.116.14:4190 , piece_id=CZGKNHBIJP7HZIC65I6ESRFRQLBKTEAQJVVUQHHZHFO7TNNROJ4A, seg_pos={0 0}, upload=301225]: Done uploading piece. err: <nil>
Jul 9 01:42:05.141496679 (376h31m39.912138074s) [noise=true , address=104.36.82.8:4969 , piece_id=6IKLPKOPP2HBAOC5FAI5IE5SNW2B2UWNWT3Q6S4WWKGDLRCARVWA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 01:42:05.141543177 (376h31m39.912184512s) [noise=true , address=104.36.82.14:4222 , piece_id=FU4BHM26LXD5XUTT333YO7ETORBXWP6QWCLUUJI2MBMPWLEFMGAA, seg_pos={0 0}, upload=301225]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 188.166.225.170:60278->104.36.82.14:4222: use of closed network connection; write tcp 188.166.225.170:60278->104.36.82.14:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60278->104.36.82.14:4222: use of closed network connection
Jul 9 01:42:05.141602951 (376h31m39.912244286s) [noise=true , address=104.36.82.5:4840 , piece_id=QKGEMZVR5P3B6ARIGS4VQDSWGAOVC5DU65Y3VLXUY2MAYVKDUQCQ, seg_pos={0 0}, upload=301225]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: write tcp 188.166.225.170:35250->104.36.82.5:4840: use of closed network connection; write tcp 188.166.225.170:35250->104.36.82.5:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35250->104.36.82.5:4840: use of closed network connection
Jul 9 01:42:05.141829285 (376h31m39.91247062s) [noise=true , address=104.36.82.11:4114 , piece_id=5ISRL5WJST2SCUOPBB7KX53T3PKO2CH5X7QKCPGSKSDO3KA5BYHQ, seg_pos={0 0}, upload=301225]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 188.166.225.170:54484->104.36.82.11:4114: use of closed network connection; write tcp 188.166.225.170:54484->104.36.82.11:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54484->104.36.82.11:4114: use of closed network connection
Jul 9 01:42:05.141852700 (376h31m39.912494025s) [noise=true , address=104.36.82.2:4730 , piece_id=EVW56CTEQD7BPUSYS2U5NXO4VRG6YZVEITV6OU5XKT3AU3MRWRYQ, seg_pos={0 0}, upload=301225]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 188.166.225.170:59748->104.36.82.2:4730: use of closed network connection; write tcp 188.166.225.170:59748->104.36.82.2:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59748->104.36.82.2:4730: use of closed network connection
Jul 9 01:42:05.141879792 (376h31m39.912521127s) [seg_pos={0 0}, upload=301225]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 9 01:42:05.366924712 (376h31m40.137566037s) [upload=301225]: Done uploading segments...
Jul 9 01:42:05.366949890 (376h31m40.137591225s) [upload=301225]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 5.65382247s
core
uplink
us-select-1
|