Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [.X...X..X............X......X..X...X.X.X.X...X..............] duration:
5.940192755s
FAIL (2025-07-09 02:09:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.940213134s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [...........X......X..........X.............X...............X] duration:
1m0.230845939s
FAIL (2025-07-09 02:09:30 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
7177 (376h58m5.878188512s) [upload=301581]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:08:31.554710025 (376h58m6.32535136s) [seg_pos={0 0}, upload=301581]: Begin upload segment...
Jul 9 02:08:31.554766423 (376h58m6.325407758s) [seg_pos={0 0}, upload=301581]: Done begin upload segment.
Jul 9 02:08:31.554787664 (376h58m6.325428989s) [noise=true , address=104.36.82.40:4944 , piece_id=2DMADEN45IW5WXG6SSK5Y72S2DKZBDBY2NXRZBHWYIICLKWDZFEQ, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.554824724 (376h58m6.325466059s) [noise=true , address=77.90.31.12:4614 , piece_id=5TIDDQKRKPQE3GJ6CMJJZTU5E3LYRTXMMS63G7EAV64Q6AXFWQRA, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.555170586 (376h58m6.325811921s) [noise=true , address=45.91.116.9:4266 , piece_id=4TFY2OQQ472NIZV4LUREV2BLF75RITUK3OEF43W3KURP2AZYRGCQ, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.555483365 (376h58m6.3261247s) [noise=true , address=104.36.82.44:4889 , piece_id=SMJJSN3XNTE53GM357QFJMOR2L6EUZP5VG5LH2HLYVG7MFUZJ7AA, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.555790893 (376h58m6.326432218s) [noise=true , address=45.91.116.7:4041 , piece_id=DI4N5ADKDSKFS6D5HUHXONI63SKBM2BLLRDNTCRD5RWSSKS2W3IQ, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.556074445 (376h58m6.32671578s) [noise=true , address=77.90.31.17:4780 , piece_id=6XI3ESZNONU2SZJJ3ELUMOMZKSOQY4SBOBRPRBERSTZ2AZRKX57A, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.556202771 (376h58m6.326844106s) [noise=true , address=193.149.164.143:4693 , piece_id=ZJKXKNFUYTUKNW25NI5IY5GKBQYNHTAKN6VRLBMYTAMCYKVYXMHQ, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.556569773 (376h58m6.327211108s) [noise=true , address=104.36.82.20:4441 , piece_id=XSH7ISLZ367ZCC6XM4YN7DDRERMG3MEBFRNH5K6WLXTKIIIPO6SA, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.556904713 (376h58m6.327546048s) [noise=true , address=77.90.31.21:4886 , piece_id=MLVSZ27JZLYNG23IRDFTDMXJRTZ3IMHUOPDGDPOO42GCERLZUQOA, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.557071302 (376h58m6.327712637s) [noise=true , address=104.36.82.35:4590 , piece_id=7KGB363U5XFEOK3UJKJXJ57P7T6XBMDUI4UZKQQ3Q4XJ3TJV54XQ, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.557342130 (376h58m6.327983465s) [noise=true , address=104.36.82.17:4345 , piece_id=JEG2KMNQ4ZXAYERB65ATCCC6FAYZX5S57GEM7YPV4274LI636YSA, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.557595104 (376h58m6.328236439s) [noise=true , address=77.90.31.10:4528 , piece_id=7R3R7QEQIQNYBV7BJF3ZGCMVJFH6OMIXZQTU3KCPSJGQTVT37RRA, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.557906210 (376h58m6.328547545s) [noise=true , address=77.90.31.16:4739 , piece_id=A52Q7XZMXUFM5BTN6XYQUAML4HVR55D6IIAY6J7P3OCFTHRIXTVQ, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.558016280 (376h58m6.328657615s) [noise=true , address=104.36.82.23:4469 , piece_id=N235KELWJET4DXCIBYJGRMR4U5GFVMERXTDM3BM62KR7WBCPJJPA, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.558293511 (376h58m6.328934846s) [noise=true , address=104.36.82.41:4783 , piece_id=L73QGLG7NQJR7VC232MVNMZXZWUL75VEISACRGNMH32GT56BFBBQ, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.558573757 (376h58m6.329215082s) [noise=true , address=104.36.82.2:4757 , piece_id=MGSUR22FTPRP6A7E7DEZNQBFBPOYQBNTGT5ZNEM64FQTCPAAJCTQ, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.558925379 (376h58m6.329566724s) [noise=true , address=77.90.31.11:4588 , piece_id=HMT3XDYW2QZCZJDKLXV7ISS3JTI7OSFC3HMRKFCZO5FBUH4TP6NA, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:31.559409055 (376h58m6.33005039s) [noise=true , address=79.127.201.214:4305 , piece_id=OL3IFYRAUOVBYLWGO4QXOMDZFGQTCXZAG73F34M7SDSLZ46MNCCA, seg_pos={0 0}, upload=301581]: Uploading piece...
Jul 9 02:08:32.571736414 (376h58m7.342377739s) [noise=true , address=79.127.201.214:4305 , piece_id=OL3IFYRAUOVBYLWGO4QXOMDZFGQTCXZAG73F34M7SDSLZ46MNCCA, seg_pos={0 0}, upload=301581]: Done uploading piece. err: ecclient: upload failed (node:12HLYBv5M6NE1Njgfz4WdagyLUqPJyc9WjEgr1CfVVEcso7HnCX, address:79.127.201.214:4305): protocol: storage node overloaded, request limit: 2000
Jul 9 02:08:32.707702667 (376h58m7.478344003s) [upload=301581]: Next returned nil segment
Jul 9 02:08:32.707717586 (376h58m7.478358922s) [upload=301581]: Waiting for error group managing segments...
Jul 9 02:08:33.045788856 (376h58m7.816430191s) [noise=true , address=193.149.164.143:4693 , piece_id=ZJKXKNFUYTUKNW25NI5IY5GKBQYNHTAKN6VRLBMYTAMCYKVYXMHQ, seg_pos={0 0}, upload=301581]: Done uploading piece. err: <nil>
Jul 9 02:08:33.477787513 (376h58m8.248428848s) [noise=true , address=45.91.116.7:4041 , piece_id=DI4N5ADKDSKFS6D5HUHXONI63SKBM2BLLRDNTCRD5RWSSKS2W3IQ, seg_pos={0 0}, upload=301581]: Done uploading piece. err: <nil>
Jul 9 02:08:33.492837816 (376h58m8.263479151s) [noise=true , address=77.90.31.17:4780 , piece_id=6XI3ESZNONU2SZJJ3ELUMOMZKSOQY4SBOBRPRBERSTZ2AZRKX57A, seg_pos={0 0}, upload=301581]: Done uploading piece. err: <nil>
Jul 9 02:08:33.505858116 (376h58m8.276499451s) [noise=true , address=77.90.31.10:4528 , piece_id=7R3R7QEQIQNYBV7BJF3ZGCMVJFH6OMIXZQTU3KCPSJGQTVT37RRA, seg_pos={0 0}, upload=301581]: Done uploading piece. err: <nil>
Jul 9 02:08:33.510054294 (376h58m8.280695629s) [noise=true , address=77.90.31.12:4614 , piece_id=5TIDDQKRKPQE3GJ6CMJJZTU5E3LYRTXMMS63G7EAV64Q6AXFWQRA, seg_pos={0 0}, upload=301581]: Done uploading piece. err: <nil>
Jul 9 02:08:33.511325926 (376h58m8.281967261s) [noise=true , address=77.90.31.11:4588 , piece_id=HMT3XDYW2QZCZJDKLXV7ISS3JTI7OSFC3HMRKFCZO5FBUH4TP6NA, seg_pos={0 0}, upload=301581]: Done uploading piece. err: <nil>
Jul 9 02:08:33.526917155 (376h58m8.29755849s) [noise=true , address=77.90.31.21:4886 , piece_id=MLVSZ27JZLYNG23IRDFTDMXJRTZ3IMHUOPDGDPOO42GCERLZUQOA, seg_pos={0 0}, upload=301581]: Done uploading piece. err: <nil>
Jul 9 02:08:33.538345038 (376h58m8.308986373s) [noise=true , address=77.90.31.16:4739 , piece_id=A52Q7XZMXUFM5BTN6XYQUAML4HVR55D6IIAY6J7P3OCFTHRIXTVQ, seg_pos={0 0}, upload=301581]: Done uploading piece. err: <nil>
Jul 9 02:08:33.609285488 (376h58m8.379926823s) [noise=true , address=45.91.116.9:4266 , piece_id=4TFY2OQQ472NIZV4LUREV2BLF75RITUK3OEF43W3KURP2AZYRGCQ, seg_pos={0 0}, upload=301581]: Done uploading piece. err: <nil>
Jul 9 02:08:46.984505697 (376h58m21.755147032s) [noise=true , address=104.36.82.20:4441 , piece_id=XSH7ISLZ367ZCC6XM4YN7DDRERMG3MEBFRNH5K6WLXTKIIIPO6SA, seg_pos={0 0}, upload=301581]: Done uploading piece. err: <nil>
Jul 9 02:08:52.339907914 (376h58m27.110549249s) [noise=true , address=104.36.82.41:4783 , piece_id=L73QGLG7NQJR7VC232MVNMZXZWUL75VEISACRGNMH32GT56BFBBQ, seg_pos={0 0}, upload=301581]: Done uploading piece. err: <nil>
Jul 9 02:09:01.032804568 (376h58m35.803445903s) [noise=true , address=104.36.82.44:4889 , piece_id=SMJJSN3XNTE53GM357QFJMOR2L6EUZP5VG5LH2HLYVG7MFUZJ7AA, seg_pos={0 0}, upload=301581]: Done uploading piece. err: <nil>
Jul 9 02:09:08.673338691 (376h58m43.443980026s) [noise=true , address=104.36.82.35:4590 , piece_id=7KGB363U5XFEOK3UJKJXJ57P7T6XBMDUI4UZKQQ3Q4XJ3TJV54XQ, seg_pos={0 0}, upload=301581]: Done uploading piece. err: <nil>
Jul 9 02:09:30.410919586 (376h59m5.181560921s) [noise=true , address=104.36.82.2:4757 , piece_id=MGSUR22FTPRP6A7E7DEZNQBFBPOYQBNTGT5ZNEM64FQTCPAAJCTQ, seg_pos={0 0}, upload=301581]: Done uploading piece. err: ecclient: upload failed (node:1uVPbSLBfJdPZqh4yFcEMQTPUwbLhT4GAM813h2nK5qa9xCw9q, address:104.36.82.2:4757): protocol: expected piece hash; write tcp 188.166.225.170:43178->104.36.82.2:4757: use of closed network connection; write tcp 188.166.225.170:43178->104.36.82.2:4757: use of closed network connection; EOF
Jul 9 02:09:30.410974972 (376h59m5.181616307s) [noise=true , address=104.36.82.40:4944 , piece_id=2DMADEN45IW5WXG6SSK5Y72S2DKZBDBY2NXRZBHWYIICLKWDZFEQ, seg_pos={0 0}, upload=301581]: Done uploading piece. err: ecclient: upload failed (node:12KMFsxzp7SfMiFBuMTaWGVyJoRTMw7ZRcfEpPDv1BmbViGF6K9, address:104.36.82.40:4944): protocol: write tcp 188.166.225.170:54250->104.36.82.40:4944: use of closed network connection; write tcp 188.166.225.170:54250->104.36.82.40:4944: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54250->104.36.82.40:4944: use of closed network connection
Jul 9 02:09:30.411014277 (376h59m5.181655612s) [noise=true , address=104.36.82.23:4469 , piece_id=N235KELWJET4DXCIBYJGRMR4U5GFVMERXTDM3BM62KR7WBCPJJPA, seg_pos={0 0}, upload=301581]: Done uploading piece. err: ecclient: upload failed (node:1RWjSXyXqhgufuBewhCBgvBfkerSFeHxJZj88UzusCgiXvibQv, address:104.36.82.23:4469): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:09:30.411035838 (376h59m5.181677174s) [noise=true , address=104.36.82.17:4345 , piece_id=JEG2KMNQ4ZXAYERB65ATCCC6FAYZX5S57GEM7YPV4274LI636YSA, seg_pos={0 0}, upload=301581]: Done uploading piece. err: ecclient: upload failed (node:1ro6Z6UAC6dNRPRWMjdMTTjhg1CvmWPENs7JcqFHxHMpmz5Fnv, address:104.36.82.17:4345): protocol: write tcp 188.166.225.170:41288->104.36.82.17:4345: use of closed network connection; write tcp 188.166.225.170:41288->104.36.82.17:4345: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41288->104.36.82.17:4345: use of closed network connection
Jul 9 02:09:30.411049184 (376h59m5.181690519s) [seg_pos={0 0}, upload=301581]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jul 9 02:09:30.640189311 (376h59m5.410830686s) [upload=301581]: Done uploading segments...
Jul 9 02:09:30.640217886 (376h59m5.410859221s) [upload=301581]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 1m0.230858192s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
18 out of 60 is failed (threshold=3) [.X......XX.XX..X......XXX....XX.....XX...X.XX........X.X....] duration:
29.364243386s
FAIL (2025-07-09 02:09:20 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 29.364258896s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [..X...X..X............X......X..X...X.X.X.X...X.............] duration:
13.976334888s
FAIL (2025-07-09 02:08:54 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 13.976358723s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
18 out of 60 is failed (threshold=3) [..X......XX.XX..X......XXX....XX.....XX...X.XX........X.X...] duration:
42.884576715s
FAIL (2025-07-09 02:08:37 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 42.884619578s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [...X......................X.................X......X........] duration:
5.690140913s
FAIL (2025-07-09 02:08:24 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
nceled; EOF
Jul 9 02:00:49.060314467 (376h50m23.830955812s) [seg_pos={0 0}, upload=301481]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil>
Jul 9 02:00:49.060367628 (376h50m23.831008963s) [upload=301481]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:00:49.060371145 (376h50m23.83101248s) [upload=301481]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:00:49.357373945 (376h50m24.12801528s) [upload=301481]: Done uploading segments...
Jul 9 02:00:49.357395647 (376h50m24.128036982s) [upload=301481]: Upload finished. err: <nil>
Jul 9 02:02:01.314289870 (376h51m36.084931205s) [upload=301495]: Starting upload
Jul 9 02:02:01.314394990 (376h51m36.085036325s) [upload=301495]: Done starting upload
Jul 9 02:02:01.314484112 (376h51m36.085125457s) [upload=301495]: Uploading segments...
Jul 9 02:02:01.314699112 (376h51m36.085340457s) [upload=301495]: Got next segment. Inline: false
Jul 9 02:02:01.314747194 (376h51m36.085388529s) [upload=301495]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:02:01.314753927 (376h51m36.085395272s) [upload=301495]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:02:01.549200488 (376h51m36.319841873s) [seg_pos={0 0}, upload=301495]: Begin upload segment...
Jul 9 02:02:01.549266264 (376h51m36.319907609s) [seg_pos={0 0}, upload=301495]: Done begin upload segment.
Jul 9 02:02:01.549310128 (376h51m36.319951473s) [noise=true , address=104.36.82.7:4910 , piece_id=CPROZR2V6J7RKRZW6CNLA7P44DQVYH3VOZMYBBTEIBOL7MGINPDA, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.549336468 (376h51m36.319977823s) [noise=true , address=45.91.116.14:4190 , piece_id=ZAF4GOHABQQENBFRPE3HEMFTT474ULDEB5MZ5XDVVHFVTQID5AVA, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.549964509 (376h51m36.320605844s) [noise=true , address=104.36.82.13:4187 , piece_id=BLI6GHKTU3D5Q6FWTOS6UEPDLBDNCX2VEZCKAXIVAF32VWU7I6XQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.550253653 (376h51m36.320894988s) [noise=true , address=104.36.82.9:4006 , piece_id=XX7OUX5VBH5UVHELOP525HLHU5LA7O4B6DR6LKI7VYFWZYVAXHTQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.550518870 (376h51m36.321160225s) [noise=true , address=104.36.82.6:4874 , piece_id=UDKHL2V3C6ZYBR3FGHVIKQYJLPCOO3XLXYTSQJVLTGJ6I74IGGAA, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.550556472 (376h51m36.321197807s) [noise=true , address=45.91.116.10:4082 , piece_id=IRWOX4OQI5TB7FAT53RSX4C6DPNPUGZPPQXS6VFM2EE6FPZI4N4A, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.550837249 (376h51m36.321478584s) [noise=true , address=45.91.116.8:4048 , piece_id=UTXJQHFKWTANYVEEL2CICL5MQWVLUC4H4ZQ3IFXQD5O45RKB425A, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.551078060 (376h51m36.321719395s) [noise=true , address=45.91.116.7:4010 , piece_id=7CYJB5QPC3ZJZVGFNJKBZHOLSKPFBYMSWTJL37CQ73JIIM7LVA2A, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.551313831 (376h51m36.321955166s) [noise=true , address=77.90.31.11:4555 , piece_id=57OEP4BSHFGZQ5A7DJKJ4BNSO7ORPXPVMSN2I56YFREXE6N5RIPQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.551369758 (376h51m36.322011093s) [noise=true , address=45.91.116.11:4119 , piece_id=2TR3FC2ZYYNAPWL2GTL5EZVJW74NI5W275AYDNSZIME62TPILDBQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.551601652 (376h51m36.322242987s) [noise=true , address=104.36.82.5:4839 , piece_id=MM6CHXUZFEON5CG7GUKRLD7PGYKU3337UBU73DTHSKAKQUJUKYLQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.551803969 (376h51m36.322445304s) [noise=true , address=45.91.116.6:4973 , piece_id=2XDTRFYDWADB2U2DIT6VDWTHT6DVRPRNMCWJFW4Q7G7KXITLI6LQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.552051763 (376h51m36.322693098s) [noise=true , address=104.36.82.15:4258 , piece_id=KFL7XC6ILPLWJGYIGZVJ6X3ECS26XNNY6M4ZS3QHGXBPZG4XWILQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.552343200 (376h51m36.322984545s) [noise=true , address=45.91.116.13:4154 , piece_id=I2XLUYF36R6HKCEYHRUISRDMAPYP7CQHEXL7XB6IMG7GYO32W55Q, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.552474742 (376h51m36.323116077s) [noise=true , address=45.91.116.4:4939 , piece_id=QDXP222CYJXWF2NV5LKI7XOPMU7APMWDFQLWURFQG5BVJUNKNJBA, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.552718097 (376h51m36.323359433s) [noise=true , address=104.36.82.11:4114 , piece_id=R7MKGZKIKESIOB5CWCDO4KV2CNGN2EXTLGATSMK7Y2JMSFIB3DSA, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:03.251350348 (376h51m38.021991683s) [upload=301495]: Next returned nil segment
Jul 9 02:02:03.251367120 (376h51m38.022008455s) [upload=301495]: Waiting for error group managing segments...
Jul 9 02:02:03.536295419 (376h51m38.306936775s) [noise=true , address=104.36.82.9:4006 , piece_id=XX7OUX5VBH5UVHELOP525HLHU5LA7O4B6DR6LKI7VYFWZYVAXHTQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.585972469 (376h51m38.356613794s) [noise=true , address=45.91.116.8:4048 , piece_id=UTXJQHFKWTANYVEEL2CICL5MQWVLUC4H4ZQ3IFXQD5O45RKB425A, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.612355463 (376h51m38.382996798s) [noise=true , address=45.91.116.11:4119 , piece_id=2TR3FC2ZYYNAPWL2GTL5EZVJW74NI5W275AYDNSZIME62TPILDBQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.616140875 (376h51m38.38678221s) [noise=true , address=45.91.116.4:4939 , piece_id=QDXP222CYJXWF2NV5LKI7XOPMU7APMWDFQLWURFQG5BVJUNKNJBA, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.625063725 (376h51m38.395705061s) [noise=true , address=45.91.116.13:4154 , piece_id=I2XLUYF36R6HKCEYHRUISRDMAPYP7CQHEXL7XB6IMG7GYO32W55Q, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.627327436 (376h51m38.397968771s) [noise=true , address=77.90.31.11:4555 , piece_id=57OEP4BSHFGZQ5A7DJKJ4BNSO7ORPXPVMSN2I56YFREXE6N5RIPQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.680079459 (376h51m38.450720804s) [noise=true , address=45.91.116.14:4190 , piece_id=ZAF4GOHABQQENBFRPE3HEMFTT474ULDEB5MZ5XDVVHFVTQID5AVA, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.682275680 (376h51m38.452917015s) [noise=true , address=45.91.116.10:4082 , piece_id=IRWOX4OQI5TB7FAT53RSX4C6DPNPUGZPPQXS6VFM2EE6FPZI4N4A, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.693418348 (376h51m38.464059684s) [noise=true , address=45.91.116.6:4973 , piece_id=2XDTRFYDWADB2U2DIT6VDWTHT6DVRPRNMCWJFW4Q7G7KXITLI6LQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.714112589 (376h51m38.484753924s) [noise=true , address=45.91.116.7:4010 , piece_id=7CYJB5QPC3ZJZVGFNJKBZHOLSKPFBYMSWTJL37CQ73JIIM7LVA2A, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:24.743005877 (376h51m59.513647202s) [noise=true , address=104.36.82.6:4874 , piece_id=UDKHL2V3C6ZYBR3FGHVIKQYJLPCOO3XLXYTSQJVLTGJ6I74IGGAA, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:28.578275204 (376h52m3.348916539s) [noise=true , address=104.36.82.13:4187 , piece_id=BLI6GHKTU3D5Q6FWTOS6UEPDLBDNCX2VEZCKAXIVAF32VWU7I6XQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:30.664723094 (376h52m5.435364439s) [noise=true , address=104.36.82.11:4114 , piece_id=R7MKGZKIKESIOB5CWCDO4KV2CNGN2EXTLGATSMK7Y2JMSFIB3DSA, seg_pos={0 0}, upload=301495]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:02:30.665014852 (376h52m5.435656188s) [noise=true , address=104.36.82.5:4839 , piece_id=MM6CHXUZFEON5CG7GUKRLD7PGYKU3337UBU73DTHSKAKQUJUKYLQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 188.166.225.170:42438->104.36.82.5:4839: use of closed network connection; write tcp 188.166.225.170:42438->104.36.82.5:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42438->104.36.82.5:4839: use of closed network connection
Jul 9 02:02:30.665113180 (376h52m5.435754525s) [noise=true , address=104.36.82.7:4910 , piece_id=CPROZR2V6J7RKRZW6CNLA7P44DQVYH3VOZMYBBTEIBOL7MGINPDA, seg_pos={0 0}, upload=301495]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: write tcp 188.166.225.170:38368->104.36.82.7:4910: use of closed network connection; write tcp 188.166.225.170:38368->104.36.82.7:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38368->104.36.82.7:4910: use of closed network connection
Jul 9 02:02:30.665314215 (376h52m5.43595555s) [noise=true , address=104.36.82.15:4258 , piece_id=KFL7XC6ILPLWJGYIGZVJ6X3ECS26XNNY6M4ZS3QHGXBPZG4XWILQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 188.166.225.170:50202->104.36.82.15:4258: use of closed network connection; context deadline exceeded
Jul 9 02:02:30.665353560 (376h52m5.435994895s) [seg_pos={0 0}, upload=301495]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 02:02:30.900452118 (376h52m5.671093453s) [upload=301495]: Done uploading segments...
Jul 9 02:02:30.900486934 (376h52m5.67112827s) [upload=301495]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.690167081s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
12 out of 60 is failed (threshold=3) [X..X...X..X............X......X..X...X.X.X.X...X............] duration:
9.528807289s
FAIL (2025-07-09 02:08:21 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 9.528836173s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [............X......X..........X.............X...............] duration:
34.405783824s
FAIL (2025-07-09 02:07:53 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
hRequestItem_SegmentBegin
Jul 9 01:56:00.629444016 (376h45m35.400085361s) [seg_pos={0 0}, upload=301417]: Begin upload segment...
Jul 9 01:56:00.629512196 (376h45m35.400153531s) [seg_pos={0 0}, upload=301417]: Done begin upload segment.
Jul 9 01:56:00.629572552 (376h45m35.400213887s) [noise=true , address=104.36.82.22:4532 , piece_id=3BRLUKXPLFBXIPKURV3BBSTA562UWWQBZ75HNYXEX75E74FKJIGQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.629579284 (376h45m35.400220619s) [noise=true , address=77.90.31.10:4539 , piece_id=OI7V6HU5WMYZSQCS5PAN6OUCAJY4UGUZVTESMLB4LHOQJVMKJLHQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.629690497 (376h45m35.400331832s) [noise=true , address=104.36.82.17:4333 , piece_id=TPALHRSYWC3XKW77CPA36EZOOJHPLUEN4V6DWFN5NHLIAFSN5KRQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630082818 (376h45m35.400724153s) [noise=true , address=104.36.82.6:4880 , piece_id=NKXAOUHYMQ4WR6V6ZGQOKRZWLAGI44AF4SNUR4OKUNJNICD4UUNQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630158242 (376h45m35.400799577s) [noise=true , address=79.127.201.214:4308 , piece_id=V4LWWG3RM4JTLZGNZ3XJDHEOZO7XW5R7BHQQJ6Z4Y47PZCEZHN6A, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630428680 (376h45m35.401070025s) [noise=true , address=77.90.31.12:4615 , piece_id=GLIW3756SYGXSX44VFMMDNYVEJPS6O4ZBNS53OJRV67QNGMGLPUA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630538129 (376h45m35.401179464s) [noise=true , address=104.36.82.60:4364 , piece_id=7WZJU4UIO4FLPUNWUH2PVIIH3ZWWVZCP2D2UUSCNBCIS727VVWPA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630862820 (376h45m35.401504165s) [noise=true , address=104.36.82.38:4708 , piece_id=RDIWOSET3Q2JKHRQ7HVUSVMNILHVEDQ7XYDL5DMWK37Q7YTRADQA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.631210084 (376h45m35.401851419s) [noise=true , address=104.36.82.21:4508 , piece_id=VVX5U234EZZNCZWIVYJVYBFDYG2SCKWS5HT5GM4H7CNNJQXDHT3Q, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.631555495 (376h45m35.40219684s) [noise=true , address=77.90.31.16:4743 , piece_id=2VSOXKLN6Q434ET7JYLRSU6BW7PIKNZ43MZOQ4HQL6MDP3WUJOZA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.631989165 (376h45m35.4026305s) [noise=true , address=104.36.82.23:4473 , piece_id=V3RAX7LZBU744EQQXDEG6Z25UUGQUFSSQCWCXNZKNGCTHD375GCA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.632258450 (376h45m35.402899785s) [noise=true , address=77.90.31.19:4850 , piece_id=DNYDFQHORBVHQTPKFLWEGQF6QRLE6IIE6YGTRP6SS63CPWVKRFFA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.632521984 (376h45m35.403163309s) [noise=true , address=45.91.116.5:4322 , piece_id=QMOE7Z74V4HHQO7LIN6KAJHEUSDKIAVC4XLFH3AOJMC2ILP6I4FA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.632835304 (376h45m35.403476649s) [noise=true , address=104.36.82.3:4791 , piece_id=NG7KRWBROBZWNP63A4734KWSRNGFVGLLXFZ5PGXZDJXFDTZNCZLQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.633120529 (376h45m35.403761864s) [noise=true , address=77.90.31.14:4692 , piece_id=NHO2352W7MMVGRS3MNPWSEFHWBOK73SQ6JQJEWZ5TV5UDFY44H5Q, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.636119667 (376h45m35.406761002s) [noise=true , address=104.36.82.2:4757 , piece_id=JMJIECO7WCFHWASHRCTEAG6VCU4S25SEETORWGJJQ63VAW533YJA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.636833744 (376h45m35.407475079s) [noise=true , address=77.90.31.15:4712 , piece_id=3DN3NPUYRN4AIBYBV2UCGE4TTYXD2WJRLHDMQEFIU36HBTYDOJ5Q, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.637112587 (376h45m35.407753922s) [noise=true , address=77.90.31.13:4643 , piece_id=PO24YVCOVVNORI7NRD3CDUC4ZPRULAXUESGBEDHJZYYDJ2ZUPQDA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:02.137197163 (376h45m36.907838498s) [upload=301417]: Next returned nil segment
Jul 9 01:56:02.137214166 (376h45m36.907855501s) [upload=301417]: Waiting for error group managing segments...
Jul 9 01:56:02.521317983 (376h45m37.291959318s) [noise=true , address=79.127.201.214:4308 , piece_id=V4LWWG3RM4JTLZGNZ3XJDHEOZO7XW5R7BHQQJ6Z4Y47PZCEZHN6A, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.567664820 (376h45m37.338306156s) [noise=true , address=77.90.31.14:4692 , piece_id=NHO2352W7MMVGRS3MNPWSEFHWBOK73SQ6JQJEWZ5TV5UDFY44H5Q, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.568760968 (376h45m37.339402303s) [noise=true , address=77.90.31.10:4539 , piece_id=OI7V6HU5WMYZSQCS5PAN6OUCAJY4UGUZVTESMLB4LHOQJVMKJLHQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.573315612 (376h45m37.343956947s) [noise=true , address=77.90.31.15:4712 , piece_id=3DN3NPUYRN4AIBYBV2UCGE4TTYXD2WJRLHDMQEFIU36HBTYDOJ5Q, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.577761728 (376h45m37.348403053s) [noise=true , address=77.90.31.19:4850 , piece_id=DNYDFQHORBVHQTPKFLWEGQF6QRLE6IIE6YGTRP6SS63CPWVKRFFA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.579676210 (376h45m37.350317545s) [noise=true , address=77.90.31.12:4615 , piece_id=GLIW3756SYGXSX44VFMMDNYVEJPS6O4ZBNS53OJRV67QNGMGLPUA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.595175322 (376h45m37.365816657s) [noise=true , address=77.90.31.16:4743 , piece_id=2VSOXKLN6Q434ET7JYLRSU6BW7PIKNZ43MZOQ4HQL6MDP3WUJOZA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.601273909 (376h45m37.371915244s) [noise=true , address=77.90.31.13:4643 , piece_id=PO24YVCOVVNORI7NRD3CDUC4ZPRULAXUESGBEDHJZYYDJ2ZUPQDA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:07.782538378 (376h45m42.553179714s) [noise=true , address=45.91.116.5:4322 , piece_id=QMOE7Z74V4HHQO7LIN6KAJHEUSDKIAVC4XLFH3AOJMC2ILP6I4FA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:22.662088184 (376h45m57.432729519s) [noise=true , address=104.36.82.17:4333 , piece_id=TPALHRSYWC3XKW77CPA36EZOOJHPLUEN4V6DWFN5NHLIAFSN5KRQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:24.053592013 (376h45m58.824233348s) [noise=true , address=104.36.82.2:4757 , piece_id=JMJIECO7WCFHWASHRCTEAG6VCU4S25SEETORWGJJQ63VAW533YJA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:44.900784468 (376h46m19.671425793s) [noise=true , address=104.36.82.23:4473 , piece_id=V3RAX7LZBU744EQQXDEG6Z25UUGQUFSSQCWCXNZKNGCTHD375GCA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:47.726372911 (376h46m22.497014236s) [noise=true , address=104.36.82.60:4364 , piece_id=7WZJU4UIO4FLPUNWUH2PVIIH3ZWWVZCP2D2UUSCNBCIS727VVWPA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:56.047624046 (376h46m30.818265381s) [noise=true , address=104.36.82.3:4791 , piece_id=NG7KRWBROBZWNP63A4734KWSRNGFVGLLXFZ5PGXZDJXFDTZNCZLQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:56.047644345 (376h46m30.81828568s) [seg_pos={0 0}, upload=301417]: Segment reached optimal threshold of 14 pieces.
Jul 9 01:56:56.047718136 (376h46m30.818359471s) [noise=true , address=104.36.82.38:4708 , piece_id=RDIWOSET3Q2JKHRQ7HVUSVMNILHVEDQ7XYDL5DMWK37Q7YTRADQA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BdJtXzvjMoNoQUjtHETyADPcZBTzcFzH8YDd651t7eTiFb1wf): protocol: expected piece hash; context canceled; EOF
Jul 9 01:56:56.047756840 (376h46m30.818398175s) [noise=true , address=104.36.82.22:4532 , piece_id=3BRLUKXPLFBXIPKURV3BBSTA562UWWQBZ75HNYXEX75E74FKJIGQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cHUuyc42zpw5k9Lz5Qo9AW9DMvKFPxRNdKZDZtjFaeyzT4Nkp): protocol: expected piece hash; context canceled; EOF
Jul 9 01:56:56.047948587 (376h46m30.818589922s) [noise=true , address=104.36.82.21:4508 , piece_id=VVX5U234EZZNCZWIVYJVYBFDYG2SCKWS5HT5GM4H7CNNJQXDHT3Q, seg_pos={0 0}, upload=301417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1269NRnqpabU6H8q9vSCmE8yMzznD1L7dyZFmzCWhtzJaTjbXg3): protocol: write tcp 188.166.225.170:60172->104.36.82.21:4508: use of closed network connection; write tcp 188.166.225.170:60172->104.36.82.21:4508: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60172->104.36.82.21:4508: use of closed network connection
Jul 9 01:56:56.047987552 (376h46m30.818628887s) [noise=true , address=104.36.82.6:4880 , piece_id=NKXAOUHYMQ4WR6V6ZGQOKRZWLAGI44AF4SNUR4OKUNJNICD4UUNQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125NsQhq7kRmTjKS8nuc51PZddApFEi5C38FzC5pEX2QrWYKzSW): protocol: write tcp 188.166.225.170:38944->104.36.82.6:4880: use of closed network connection; write tcp 188.166.225.170:38944->104.36.82.6:4880: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38944->104.36.82.6:4880: use of closed network connection
Jul 9 01:56:56.047996138 (376h46m30.818637463s) [seg_pos={0 0}, upload=301417]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 01:56:56.048046284 (376h46m30.818687619s) [upload=301417]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 01:56:56.048050091 (376h46m30.818691416s) [upload=301417]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 01:56:56.339173192 (376h46m31.109814527s) [upload=301417]: Done uploading segments...
Jul 9 01:56:56.339190416 (376h46m31.109831751s) [upload=301417]: Upload finished. err: <nil>
Duration: 34.405806959s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
18 out of 60 is failed (threshold=3) [...X......XX.XX..X......XXX....XX.....XX...X.XX........X.X..] duration:
36.765325027s
FAIL (2025-07-09 02:07:31 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 36.765345346s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [....X......................X.................X......X.......] duration:
5.702014777s
FAIL (2025-07-09 02:07:22 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
nceled; EOF
Jul 9 02:00:49.060314467 (376h50m23.830955812s) [seg_pos={0 0}, upload=301481]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil>
Jul 9 02:00:49.060367628 (376h50m23.831008963s) [upload=301481]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:00:49.060371145 (376h50m23.83101248s) [upload=301481]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:00:49.357373945 (376h50m24.12801528s) [upload=301481]: Done uploading segments...
Jul 9 02:00:49.357395647 (376h50m24.128036982s) [upload=301481]: Upload finished. err: <nil>
Jul 9 02:02:01.314289870 (376h51m36.084931205s) [upload=301495]: Starting upload
Jul 9 02:02:01.314394990 (376h51m36.085036325s) [upload=301495]: Done starting upload
Jul 9 02:02:01.314484112 (376h51m36.085125457s) [upload=301495]: Uploading segments...
Jul 9 02:02:01.314699112 (376h51m36.085340457s) [upload=301495]: Got next segment. Inline: false
Jul 9 02:02:01.314747194 (376h51m36.085388529s) [upload=301495]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:02:01.314753927 (376h51m36.085395272s) [upload=301495]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:02:01.549200488 (376h51m36.319841873s) [seg_pos={0 0}, upload=301495]: Begin upload segment...
Jul 9 02:02:01.549266264 (376h51m36.319907609s) [seg_pos={0 0}, upload=301495]: Done begin upload segment.
Jul 9 02:02:01.549310128 (376h51m36.319951473s) [noise=true , address=104.36.82.7:4910 , piece_id=CPROZR2V6J7RKRZW6CNLA7P44DQVYH3VOZMYBBTEIBOL7MGINPDA, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.549336468 (376h51m36.319977823s) [noise=true , address=45.91.116.14:4190 , piece_id=ZAF4GOHABQQENBFRPE3HEMFTT474ULDEB5MZ5XDVVHFVTQID5AVA, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.549964509 (376h51m36.320605844s) [noise=true , address=104.36.82.13:4187 , piece_id=BLI6GHKTU3D5Q6FWTOS6UEPDLBDNCX2VEZCKAXIVAF32VWU7I6XQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.550253653 (376h51m36.320894988s) [noise=true , address=104.36.82.9:4006 , piece_id=XX7OUX5VBH5UVHELOP525HLHU5LA7O4B6DR6LKI7VYFWZYVAXHTQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.550518870 (376h51m36.321160225s) [noise=true , address=104.36.82.6:4874 , piece_id=UDKHL2V3C6ZYBR3FGHVIKQYJLPCOO3XLXYTSQJVLTGJ6I74IGGAA, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.550556472 (376h51m36.321197807s) [noise=true , address=45.91.116.10:4082 , piece_id=IRWOX4OQI5TB7FAT53RSX4C6DPNPUGZPPQXS6VFM2EE6FPZI4N4A, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.550837249 (376h51m36.321478584s) [noise=true , address=45.91.116.8:4048 , piece_id=UTXJQHFKWTANYVEEL2CICL5MQWVLUC4H4ZQ3IFXQD5O45RKB425A, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.551078060 (376h51m36.321719395s) [noise=true , address=45.91.116.7:4010 , piece_id=7CYJB5QPC3ZJZVGFNJKBZHOLSKPFBYMSWTJL37CQ73JIIM7LVA2A, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.551313831 (376h51m36.321955166s) [noise=true , address=77.90.31.11:4555 , piece_id=57OEP4BSHFGZQ5A7DJKJ4BNSO7ORPXPVMSN2I56YFREXE6N5RIPQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.551369758 (376h51m36.322011093s) [noise=true , address=45.91.116.11:4119 , piece_id=2TR3FC2ZYYNAPWL2GTL5EZVJW74NI5W275AYDNSZIME62TPILDBQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.551601652 (376h51m36.322242987s) [noise=true , address=104.36.82.5:4839 , piece_id=MM6CHXUZFEON5CG7GUKRLD7PGYKU3337UBU73DTHSKAKQUJUKYLQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.551803969 (376h51m36.322445304s) [noise=true , address=45.91.116.6:4973 , piece_id=2XDTRFYDWADB2U2DIT6VDWTHT6DVRPRNMCWJFW4Q7G7KXITLI6LQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.552051763 (376h51m36.322693098s) [noise=true , address=104.36.82.15:4258 , piece_id=KFL7XC6ILPLWJGYIGZVJ6X3ECS26XNNY6M4ZS3QHGXBPZG4XWILQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.552343200 (376h51m36.322984545s) [noise=true , address=45.91.116.13:4154 , piece_id=I2XLUYF36R6HKCEYHRUISRDMAPYP7CQHEXL7XB6IMG7GYO32W55Q, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.552474742 (376h51m36.323116077s) [noise=true , address=45.91.116.4:4939 , piece_id=QDXP222CYJXWF2NV5LKI7XOPMU7APMWDFQLWURFQG5BVJUNKNJBA, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.552718097 (376h51m36.323359433s) [noise=true , address=104.36.82.11:4114 , piece_id=R7MKGZKIKESIOB5CWCDO4KV2CNGN2EXTLGATSMK7Y2JMSFIB3DSA, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:03.251350348 (376h51m38.021991683s) [upload=301495]: Next returned nil segment
Jul 9 02:02:03.251367120 (376h51m38.022008455s) [upload=301495]: Waiting for error group managing segments...
Jul 9 02:02:03.536295419 (376h51m38.306936775s) [noise=true , address=104.36.82.9:4006 , piece_id=XX7OUX5VBH5UVHELOP525HLHU5LA7O4B6DR6LKI7VYFWZYVAXHTQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.585972469 (376h51m38.356613794s) [noise=true , address=45.91.116.8:4048 , piece_id=UTXJQHFKWTANYVEEL2CICL5MQWVLUC4H4ZQ3IFXQD5O45RKB425A, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.612355463 (376h51m38.382996798s) [noise=true , address=45.91.116.11:4119 , piece_id=2TR3FC2ZYYNAPWL2GTL5EZVJW74NI5W275AYDNSZIME62TPILDBQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.616140875 (376h51m38.38678221s) [noise=true , address=45.91.116.4:4939 , piece_id=QDXP222CYJXWF2NV5LKI7XOPMU7APMWDFQLWURFQG5BVJUNKNJBA, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.625063725 (376h51m38.395705061s) [noise=true , address=45.91.116.13:4154 , piece_id=I2XLUYF36R6HKCEYHRUISRDMAPYP7CQHEXL7XB6IMG7GYO32W55Q, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.627327436 (376h51m38.397968771s) [noise=true , address=77.90.31.11:4555 , piece_id=57OEP4BSHFGZQ5A7DJKJ4BNSO7ORPXPVMSN2I56YFREXE6N5RIPQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.680079459 (376h51m38.450720804s) [noise=true , address=45.91.116.14:4190 , piece_id=ZAF4GOHABQQENBFRPE3HEMFTT474ULDEB5MZ5XDVVHFVTQID5AVA, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.682275680 (376h51m38.452917015s) [noise=true , address=45.91.116.10:4082 , piece_id=IRWOX4OQI5TB7FAT53RSX4C6DPNPUGZPPQXS6VFM2EE6FPZI4N4A, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.693418348 (376h51m38.464059684s) [noise=true , address=45.91.116.6:4973 , piece_id=2XDTRFYDWADB2U2DIT6VDWTHT6DVRPRNMCWJFW4Q7G7KXITLI6LQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.714112589 (376h51m38.484753924s) [noise=true , address=45.91.116.7:4010 , piece_id=7CYJB5QPC3ZJZVGFNJKBZHOLSKPFBYMSWTJL37CQ73JIIM7LVA2A, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:24.743005877 (376h51m59.513647202s) [noise=true , address=104.36.82.6:4874 , piece_id=UDKHL2V3C6ZYBR3FGHVIKQYJLPCOO3XLXYTSQJVLTGJ6I74IGGAA, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:28.578275204 (376h52m3.348916539s) [noise=true , address=104.36.82.13:4187 , piece_id=BLI6GHKTU3D5Q6FWTOS6UEPDLBDNCX2VEZCKAXIVAF32VWU7I6XQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:30.664723094 (376h52m5.435364439s) [noise=true , address=104.36.82.11:4114 , piece_id=R7MKGZKIKESIOB5CWCDO4KV2CNGN2EXTLGATSMK7Y2JMSFIB3DSA, seg_pos={0 0}, upload=301495]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:02:30.665014852 (376h52m5.435656188s) [noise=true , address=104.36.82.5:4839 , piece_id=MM6CHXUZFEON5CG7GUKRLD7PGYKU3337UBU73DTHSKAKQUJUKYLQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 188.166.225.170:42438->104.36.82.5:4839: use of closed network connection; write tcp 188.166.225.170:42438->104.36.82.5:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42438->104.36.82.5:4839: use of closed network connection
Jul 9 02:02:30.665113180 (376h52m5.435754525s) [noise=true , address=104.36.82.7:4910 , piece_id=CPROZR2V6J7RKRZW6CNLA7P44DQVYH3VOZMYBBTEIBOL7MGINPDA, seg_pos={0 0}, upload=301495]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: write tcp 188.166.225.170:38368->104.36.82.7:4910: use of closed network connection; write tcp 188.166.225.170:38368->104.36.82.7:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38368->104.36.82.7:4910: use of closed network connection
Jul 9 02:02:30.665314215 (376h52m5.43595555s) [noise=true , address=104.36.82.15:4258 , piece_id=KFL7XC6ILPLWJGYIGZVJ6X3ECS26XNNY6M4ZS3QHGXBPZG4XWILQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 188.166.225.170:50202->104.36.82.15:4258: use of closed network connection; context deadline exceeded
Jul 9 02:02:30.665353560 (376h52m5.435994895s) [seg_pos={0 0}, upload=301495]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 02:02:30.900452118 (376h52m5.671093453s) [upload=301495]: Done uploading segments...
Jul 9 02:02:30.900486934 (376h52m5.67112827s) [upload=301495]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.702040697s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
12 out of 60 is failed (threshold=3) [.X..X...X..X............X......X..X...X.X.X.X...X...........] duration:
8.869038877s
FAIL (2025-07-09 02:07:13 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 8.869058925s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [.............X......X..........X.............X..............] duration:
26.21498432s
FAIL (2025-07-09 02:07:05 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
hRequestItem_SegmentBegin
Jul 9 01:56:00.629444016 (376h45m35.400085361s) [seg_pos={0 0}, upload=301417]: Begin upload segment...
Jul 9 01:56:00.629512196 (376h45m35.400153531s) [seg_pos={0 0}, upload=301417]: Done begin upload segment.
Jul 9 01:56:00.629572552 (376h45m35.400213887s) [noise=true , address=104.36.82.22:4532 , piece_id=3BRLUKXPLFBXIPKURV3BBSTA562UWWQBZ75HNYXEX75E74FKJIGQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.629579284 (376h45m35.400220619s) [noise=true , address=77.90.31.10:4539 , piece_id=OI7V6HU5WMYZSQCS5PAN6OUCAJY4UGUZVTESMLB4LHOQJVMKJLHQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.629690497 (376h45m35.400331832s) [noise=true , address=104.36.82.17:4333 , piece_id=TPALHRSYWC3XKW77CPA36EZOOJHPLUEN4V6DWFN5NHLIAFSN5KRQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630082818 (376h45m35.400724153s) [noise=true , address=104.36.82.6:4880 , piece_id=NKXAOUHYMQ4WR6V6ZGQOKRZWLAGI44AF4SNUR4OKUNJNICD4UUNQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630158242 (376h45m35.400799577s) [noise=true , address=79.127.201.214:4308 , piece_id=V4LWWG3RM4JTLZGNZ3XJDHEOZO7XW5R7BHQQJ6Z4Y47PZCEZHN6A, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630428680 (376h45m35.401070025s) [noise=true , address=77.90.31.12:4615 , piece_id=GLIW3756SYGXSX44VFMMDNYVEJPS6O4ZBNS53OJRV67QNGMGLPUA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630538129 (376h45m35.401179464s) [noise=true , address=104.36.82.60:4364 , piece_id=7WZJU4UIO4FLPUNWUH2PVIIH3ZWWVZCP2D2UUSCNBCIS727VVWPA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630862820 (376h45m35.401504165s) [noise=true , address=104.36.82.38:4708 , piece_id=RDIWOSET3Q2JKHRQ7HVUSVMNILHVEDQ7XYDL5DMWK37Q7YTRADQA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.631210084 (376h45m35.401851419s) [noise=true , address=104.36.82.21:4508 , piece_id=VVX5U234EZZNCZWIVYJVYBFDYG2SCKWS5HT5GM4H7CNNJQXDHT3Q, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.631555495 (376h45m35.40219684s) [noise=true , address=77.90.31.16:4743 , piece_id=2VSOXKLN6Q434ET7JYLRSU6BW7PIKNZ43MZOQ4HQL6MDP3WUJOZA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.631989165 (376h45m35.4026305s) [noise=true , address=104.36.82.23:4473 , piece_id=V3RAX7LZBU744EQQXDEG6Z25UUGQUFSSQCWCXNZKNGCTHD375GCA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.632258450 (376h45m35.402899785s) [noise=true , address=77.90.31.19:4850 , piece_id=DNYDFQHORBVHQTPKFLWEGQF6QRLE6IIE6YGTRP6SS63CPWVKRFFA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.632521984 (376h45m35.403163309s) [noise=true , address=45.91.116.5:4322 , piece_id=QMOE7Z74V4HHQO7LIN6KAJHEUSDKIAVC4XLFH3AOJMC2ILP6I4FA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.632835304 (376h45m35.403476649s) [noise=true , address=104.36.82.3:4791 , piece_id=NG7KRWBROBZWNP63A4734KWSRNGFVGLLXFZ5PGXZDJXFDTZNCZLQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.633120529 (376h45m35.403761864s) [noise=true , address=77.90.31.14:4692 , piece_id=NHO2352W7MMVGRS3MNPWSEFHWBOK73SQ6JQJEWZ5TV5UDFY44H5Q, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.636119667 (376h45m35.406761002s) [noise=true , address=104.36.82.2:4757 , piece_id=JMJIECO7WCFHWASHRCTEAG6VCU4S25SEETORWGJJQ63VAW533YJA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.636833744 (376h45m35.407475079s) [noise=true , address=77.90.31.15:4712 , piece_id=3DN3NPUYRN4AIBYBV2UCGE4TTYXD2WJRLHDMQEFIU36HBTYDOJ5Q, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.637112587 (376h45m35.407753922s) [noise=true , address=77.90.31.13:4643 , piece_id=PO24YVCOVVNORI7NRD3CDUC4ZPRULAXUESGBEDHJZYYDJ2ZUPQDA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:02.137197163 (376h45m36.907838498s) [upload=301417]: Next returned nil segment
Jul 9 01:56:02.137214166 (376h45m36.907855501s) [upload=301417]: Waiting for error group managing segments...
Jul 9 01:56:02.521317983 (376h45m37.291959318s) [noise=true , address=79.127.201.214:4308 , piece_id=V4LWWG3RM4JTLZGNZ3XJDHEOZO7XW5R7BHQQJ6Z4Y47PZCEZHN6A, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.567664820 (376h45m37.338306156s) [noise=true , address=77.90.31.14:4692 , piece_id=NHO2352W7MMVGRS3MNPWSEFHWBOK73SQ6JQJEWZ5TV5UDFY44H5Q, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.568760968 (376h45m37.339402303s) [noise=true , address=77.90.31.10:4539 , piece_id=OI7V6HU5WMYZSQCS5PAN6OUCAJY4UGUZVTESMLB4LHOQJVMKJLHQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.573315612 (376h45m37.343956947s) [noise=true , address=77.90.31.15:4712 , piece_id=3DN3NPUYRN4AIBYBV2UCGE4TTYXD2WJRLHDMQEFIU36HBTYDOJ5Q, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.577761728 (376h45m37.348403053s) [noise=true , address=77.90.31.19:4850 , piece_id=DNYDFQHORBVHQTPKFLWEGQF6QRLE6IIE6YGTRP6SS63CPWVKRFFA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.579676210 (376h45m37.350317545s) [noise=true , address=77.90.31.12:4615 , piece_id=GLIW3756SYGXSX44VFMMDNYVEJPS6O4ZBNS53OJRV67QNGMGLPUA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.595175322 (376h45m37.365816657s) [noise=true , address=77.90.31.16:4743 , piece_id=2VSOXKLN6Q434ET7JYLRSU6BW7PIKNZ43MZOQ4HQL6MDP3WUJOZA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.601273909 (376h45m37.371915244s) [noise=true , address=77.90.31.13:4643 , piece_id=PO24YVCOVVNORI7NRD3CDUC4ZPRULAXUESGBEDHJZYYDJ2ZUPQDA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:07.782538378 (376h45m42.553179714s) [noise=true , address=45.91.116.5:4322 , piece_id=QMOE7Z74V4HHQO7LIN6KAJHEUSDKIAVC4XLFH3AOJMC2ILP6I4FA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:22.662088184 (376h45m57.432729519s) [noise=true , address=104.36.82.17:4333 , piece_id=TPALHRSYWC3XKW77CPA36EZOOJHPLUEN4V6DWFN5NHLIAFSN5KRQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:24.053592013 (376h45m58.824233348s) [noise=true , address=104.36.82.2:4757 , piece_id=JMJIECO7WCFHWASHRCTEAG6VCU4S25SEETORWGJJQ63VAW533YJA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:44.900784468 (376h46m19.671425793s) [noise=true , address=104.36.82.23:4473 , piece_id=V3RAX7LZBU744EQQXDEG6Z25UUGQUFSSQCWCXNZKNGCTHD375GCA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:47.726372911 (376h46m22.497014236s) [noise=true , address=104.36.82.60:4364 , piece_id=7WZJU4UIO4FLPUNWUH2PVIIH3ZWWVZCP2D2UUSCNBCIS727VVWPA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:56.047624046 (376h46m30.818265381s) [noise=true , address=104.36.82.3:4791 , piece_id=NG7KRWBROBZWNP63A4734KWSRNGFVGLLXFZ5PGXZDJXFDTZNCZLQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:56.047644345 (376h46m30.81828568s) [seg_pos={0 0}, upload=301417]: Segment reached optimal threshold of 14 pieces.
Jul 9 01:56:56.047718136 (376h46m30.818359471s) [noise=true , address=104.36.82.38:4708 , piece_id=RDIWOSET3Q2JKHRQ7HVUSVMNILHVEDQ7XYDL5DMWK37Q7YTRADQA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BdJtXzvjMoNoQUjtHETyADPcZBTzcFzH8YDd651t7eTiFb1wf): protocol: expected piece hash; context canceled; EOF
Jul 9 01:56:56.047756840 (376h46m30.818398175s) [noise=true , address=104.36.82.22:4532 , piece_id=3BRLUKXPLFBXIPKURV3BBSTA562UWWQBZ75HNYXEX75E74FKJIGQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cHUuyc42zpw5k9Lz5Qo9AW9DMvKFPxRNdKZDZtjFaeyzT4Nkp): protocol: expected piece hash; context canceled; EOF
Jul 9 01:56:56.047948587 (376h46m30.818589922s) [noise=true , address=104.36.82.21:4508 , piece_id=VVX5U234EZZNCZWIVYJVYBFDYG2SCKWS5HT5GM4H7CNNJQXDHT3Q, seg_pos={0 0}, upload=301417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1269NRnqpabU6H8q9vSCmE8yMzznD1L7dyZFmzCWhtzJaTjbXg3): protocol: write tcp 188.166.225.170:60172->104.36.82.21:4508: use of closed network connection; write tcp 188.166.225.170:60172->104.36.82.21:4508: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60172->104.36.82.21:4508: use of closed network connection
Jul 9 01:56:56.047987552 (376h46m30.818628887s) [noise=true , address=104.36.82.6:4880 , piece_id=NKXAOUHYMQ4WR6V6ZGQOKRZWLAGI44AF4SNUR4OKUNJNICD4UUNQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125NsQhq7kRmTjKS8nuc51PZddApFEi5C38FzC5pEX2QrWYKzSW): protocol: write tcp 188.166.225.170:38944->104.36.82.6:4880: use of closed network connection; write tcp 188.166.225.170:38944->104.36.82.6:4880: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38944->104.36.82.6:4880: use of closed network connection
Jul 9 01:56:56.047996138 (376h46m30.818637463s) [seg_pos={0 0}, upload=301417]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 01:56:56.048046284 (376h46m30.818687619s) [upload=301417]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 01:56:56.048050091 (376h46m30.818691416s) [upload=301417]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 01:56:56.339173192 (376h46m31.109814527s) [upload=301417]: Done uploading segments...
Jul 9 01:56:56.339190416 (376h46m31.109831751s) [upload=301417]: Upload finished. err: <nil>
Duration: 26.215001013s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.....X......................X.................X......X......] duration:
5.783831203s
FAIL (2025-07-09 02:06:58 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
nceled; EOF
Jul 9 02:00:49.060314467 (376h50m23.830955812s) [seg_pos={0 0}, upload=301481]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil>
Jul 9 02:00:49.060367628 (376h50m23.831008963s) [upload=301481]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:00:49.060371145 (376h50m23.83101248s) [upload=301481]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:00:49.357373945 (376h50m24.12801528s) [upload=301481]: Done uploading segments...
Jul 9 02:00:49.357395647 (376h50m24.128036982s) [upload=301481]: Upload finished. err: <nil>
Jul 9 02:02:01.314289870 (376h51m36.084931205s) [upload=301495]: Starting upload
Jul 9 02:02:01.314394990 (376h51m36.085036325s) [upload=301495]: Done starting upload
Jul 9 02:02:01.314484112 (376h51m36.085125457s) [upload=301495]: Uploading segments...
Jul 9 02:02:01.314699112 (376h51m36.085340457s) [upload=301495]: Got next segment. Inline: false
Jul 9 02:02:01.314747194 (376h51m36.085388529s) [upload=301495]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:02:01.314753927 (376h51m36.085395272s) [upload=301495]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:02:01.549200488 (376h51m36.319841873s) [seg_pos={0 0}, upload=301495]: Begin upload segment...
Jul 9 02:02:01.549266264 (376h51m36.319907609s) [seg_pos={0 0}, upload=301495]: Done begin upload segment.
Jul 9 02:02:01.549310128 (376h51m36.319951473s) [noise=true , address=104.36.82.7:4910 , piece_id=CPROZR2V6J7RKRZW6CNLA7P44DQVYH3VOZMYBBTEIBOL7MGINPDA, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.549336468 (376h51m36.319977823s) [noise=true , address=45.91.116.14:4190 , piece_id=ZAF4GOHABQQENBFRPE3HEMFTT474ULDEB5MZ5XDVVHFVTQID5AVA, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.549964509 (376h51m36.320605844s) [noise=true , address=104.36.82.13:4187 , piece_id=BLI6GHKTU3D5Q6FWTOS6UEPDLBDNCX2VEZCKAXIVAF32VWU7I6XQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.550253653 (376h51m36.320894988s) [noise=true , address=104.36.82.9:4006 , piece_id=XX7OUX5VBH5UVHELOP525HLHU5LA7O4B6DR6LKI7VYFWZYVAXHTQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.550518870 (376h51m36.321160225s) [noise=true , address=104.36.82.6:4874 , piece_id=UDKHL2V3C6ZYBR3FGHVIKQYJLPCOO3XLXYTSQJVLTGJ6I74IGGAA, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.550556472 (376h51m36.321197807s) [noise=true , address=45.91.116.10:4082 , piece_id=IRWOX4OQI5TB7FAT53RSX4C6DPNPUGZPPQXS6VFM2EE6FPZI4N4A, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.550837249 (376h51m36.321478584s) [noise=true , address=45.91.116.8:4048 , piece_id=UTXJQHFKWTANYVEEL2CICL5MQWVLUC4H4ZQ3IFXQD5O45RKB425A, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.551078060 (376h51m36.321719395s) [noise=true , address=45.91.116.7:4010 , piece_id=7CYJB5QPC3ZJZVGFNJKBZHOLSKPFBYMSWTJL37CQ73JIIM7LVA2A, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.551313831 (376h51m36.321955166s) [noise=true , address=77.90.31.11:4555 , piece_id=57OEP4BSHFGZQ5A7DJKJ4BNSO7ORPXPVMSN2I56YFREXE6N5RIPQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.551369758 (376h51m36.322011093s) [noise=true , address=45.91.116.11:4119 , piece_id=2TR3FC2ZYYNAPWL2GTL5EZVJW74NI5W275AYDNSZIME62TPILDBQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.551601652 (376h51m36.322242987s) [noise=true , address=104.36.82.5:4839 , piece_id=MM6CHXUZFEON5CG7GUKRLD7PGYKU3337UBU73DTHSKAKQUJUKYLQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.551803969 (376h51m36.322445304s) [noise=true , address=45.91.116.6:4973 , piece_id=2XDTRFYDWADB2U2DIT6VDWTHT6DVRPRNMCWJFW4Q7G7KXITLI6LQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.552051763 (376h51m36.322693098s) [noise=true , address=104.36.82.15:4258 , piece_id=KFL7XC6ILPLWJGYIGZVJ6X3ECS26XNNY6M4ZS3QHGXBPZG4XWILQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.552343200 (376h51m36.322984545s) [noise=true , address=45.91.116.13:4154 , piece_id=I2XLUYF36R6HKCEYHRUISRDMAPYP7CQHEXL7XB6IMG7GYO32W55Q, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.552474742 (376h51m36.323116077s) [noise=true , address=45.91.116.4:4939 , piece_id=QDXP222CYJXWF2NV5LKI7XOPMU7APMWDFQLWURFQG5BVJUNKNJBA, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.552718097 (376h51m36.323359433s) [noise=true , address=104.36.82.11:4114 , piece_id=R7MKGZKIKESIOB5CWCDO4KV2CNGN2EXTLGATSMK7Y2JMSFIB3DSA, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:03.251350348 (376h51m38.021991683s) [upload=301495]: Next returned nil segment
Jul 9 02:02:03.251367120 (376h51m38.022008455s) [upload=301495]: Waiting for error group managing segments...
Jul 9 02:02:03.536295419 (376h51m38.306936775s) [noise=true , address=104.36.82.9:4006 , piece_id=XX7OUX5VBH5UVHELOP525HLHU5LA7O4B6DR6LKI7VYFWZYVAXHTQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.585972469 (376h51m38.356613794s) [noise=true , address=45.91.116.8:4048 , piece_id=UTXJQHFKWTANYVEEL2CICL5MQWVLUC4H4ZQ3IFXQD5O45RKB425A, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.612355463 (376h51m38.382996798s) [noise=true , address=45.91.116.11:4119 , piece_id=2TR3FC2ZYYNAPWL2GTL5EZVJW74NI5W275AYDNSZIME62TPILDBQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.616140875 (376h51m38.38678221s) [noise=true , address=45.91.116.4:4939 , piece_id=QDXP222CYJXWF2NV5LKI7XOPMU7APMWDFQLWURFQG5BVJUNKNJBA, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.625063725 (376h51m38.395705061s) [noise=true , address=45.91.116.13:4154 , piece_id=I2XLUYF36R6HKCEYHRUISRDMAPYP7CQHEXL7XB6IMG7GYO32W55Q, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.627327436 (376h51m38.397968771s) [noise=true , address=77.90.31.11:4555 , piece_id=57OEP4BSHFGZQ5A7DJKJ4BNSO7ORPXPVMSN2I56YFREXE6N5RIPQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.680079459 (376h51m38.450720804s) [noise=true , address=45.91.116.14:4190 , piece_id=ZAF4GOHABQQENBFRPE3HEMFTT474ULDEB5MZ5XDVVHFVTQID5AVA, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.682275680 (376h51m38.452917015s) [noise=true , address=45.91.116.10:4082 , piece_id=IRWOX4OQI5TB7FAT53RSX4C6DPNPUGZPPQXS6VFM2EE6FPZI4N4A, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.693418348 (376h51m38.464059684s) [noise=true , address=45.91.116.6:4973 , piece_id=2XDTRFYDWADB2U2DIT6VDWTHT6DVRPRNMCWJFW4Q7G7KXITLI6LQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.714112589 (376h51m38.484753924s) [noise=true , address=45.91.116.7:4010 , piece_id=7CYJB5QPC3ZJZVGFNJKBZHOLSKPFBYMSWTJL37CQ73JIIM7LVA2A, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:24.743005877 (376h51m59.513647202s) [noise=true , address=104.36.82.6:4874 , piece_id=UDKHL2V3C6ZYBR3FGHVIKQYJLPCOO3XLXYTSQJVLTGJ6I74IGGAA, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:28.578275204 (376h52m3.348916539s) [noise=true , address=104.36.82.13:4187 , piece_id=BLI6GHKTU3D5Q6FWTOS6UEPDLBDNCX2VEZCKAXIVAF32VWU7I6XQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:30.664723094 (376h52m5.435364439s) [noise=true , address=104.36.82.11:4114 , piece_id=R7MKGZKIKESIOB5CWCDO4KV2CNGN2EXTLGATSMK7Y2JMSFIB3DSA, seg_pos={0 0}, upload=301495]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:02:30.665014852 (376h52m5.435656188s) [noise=true , address=104.36.82.5:4839 , piece_id=MM6CHXUZFEON5CG7GUKRLD7PGYKU3337UBU73DTHSKAKQUJUKYLQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 188.166.225.170:42438->104.36.82.5:4839: use of closed network connection; write tcp 188.166.225.170:42438->104.36.82.5:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42438->104.36.82.5:4839: use of closed network connection
Jul 9 02:02:30.665113180 (376h52m5.435754525s) [noise=true , address=104.36.82.7:4910 , piece_id=CPROZR2V6J7RKRZW6CNLA7P44DQVYH3VOZMYBBTEIBOL7MGINPDA, seg_pos={0 0}, upload=301495]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: write tcp 188.166.225.170:38368->104.36.82.7:4910: use of closed network connection; write tcp 188.166.225.170:38368->104.36.82.7:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38368->104.36.82.7:4910: use of closed network connection
Jul 9 02:02:30.665314215 (376h52m5.43595555s) [noise=true , address=104.36.82.15:4258 , piece_id=KFL7XC6ILPLWJGYIGZVJ6X3ECS26XNNY6M4ZS3QHGXBPZG4XWILQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 188.166.225.170:50202->104.36.82.15:4258: use of closed network connection; context deadline exceeded
Jul 9 02:02:30.665353560 (376h52m5.435994895s) [seg_pos={0 0}, upload=301495]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 02:02:30.900452118 (376h52m5.671093453s) [upload=301495]: Done uploading segments...
Jul 9 02:02:30.900486934 (376h52m5.67112827s) [upload=301495]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.783854937s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
18 out of 60 is failed (threshold=3) [....X......XX.XX..X......XXX....XX.....XX...X.XX........X.X.] duration:
57.324259211s
FAIL (2025-07-09 02:06:42 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 57.324278928s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [..............X......X..........X.............X.............] duration:
50.590796748s
FAIL (2025-07-09 02:06:39 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
hRequestItem_SegmentBegin
Jul 9 01:56:00.629444016 (376h45m35.400085361s) [seg_pos={0 0}, upload=301417]: Begin upload segment...
Jul 9 01:56:00.629512196 (376h45m35.400153531s) [seg_pos={0 0}, upload=301417]: Done begin upload segment.
Jul 9 01:56:00.629572552 (376h45m35.400213887s) [noise=true , address=104.36.82.22:4532 , piece_id=3BRLUKXPLFBXIPKURV3BBSTA562UWWQBZ75HNYXEX75E74FKJIGQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.629579284 (376h45m35.400220619s) [noise=true , address=77.90.31.10:4539 , piece_id=OI7V6HU5WMYZSQCS5PAN6OUCAJY4UGUZVTESMLB4LHOQJVMKJLHQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.629690497 (376h45m35.400331832s) [noise=true , address=104.36.82.17:4333 , piece_id=TPALHRSYWC3XKW77CPA36EZOOJHPLUEN4V6DWFN5NHLIAFSN5KRQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630082818 (376h45m35.400724153s) [noise=true , address=104.36.82.6:4880 , piece_id=NKXAOUHYMQ4WR6V6ZGQOKRZWLAGI44AF4SNUR4OKUNJNICD4UUNQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630158242 (376h45m35.400799577s) [noise=true , address=79.127.201.214:4308 , piece_id=V4LWWG3RM4JTLZGNZ3XJDHEOZO7XW5R7BHQQJ6Z4Y47PZCEZHN6A, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630428680 (376h45m35.401070025s) [noise=true , address=77.90.31.12:4615 , piece_id=GLIW3756SYGXSX44VFMMDNYVEJPS6O4ZBNS53OJRV67QNGMGLPUA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630538129 (376h45m35.401179464s) [noise=true , address=104.36.82.60:4364 , piece_id=7WZJU4UIO4FLPUNWUH2PVIIH3ZWWVZCP2D2UUSCNBCIS727VVWPA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630862820 (376h45m35.401504165s) [noise=true , address=104.36.82.38:4708 , piece_id=RDIWOSET3Q2JKHRQ7HVUSVMNILHVEDQ7XYDL5DMWK37Q7YTRADQA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.631210084 (376h45m35.401851419s) [noise=true , address=104.36.82.21:4508 , piece_id=VVX5U234EZZNCZWIVYJVYBFDYG2SCKWS5HT5GM4H7CNNJQXDHT3Q, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.631555495 (376h45m35.40219684s) [noise=true , address=77.90.31.16:4743 , piece_id=2VSOXKLN6Q434ET7JYLRSU6BW7PIKNZ43MZOQ4HQL6MDP3WUJOZA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.631989165 (376h45m35.4026305s) [noise=true , address=104.36.82.23:4473 , piece_id=V3RAX7LZBU744EQQXDEG6Z25UUGQUFSSQCWCXNZKNGCTHD375GCA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.632258450 (376h45m35.402899785s) [noise=true , address=77.90.31.19:4850 , piece_id=DNYDFQHORBVHQTPKFLWEGQF6QRLE6IIE6YGTRP6SS63CPWVKRFFA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.632521984 (376h45m35.403163309s) [noise=true , address=45.91.116.5:4322 , piece_id=QMOE7Z74V4HHQO7LIN6KAJHEUSDKIAVC4XLFH3AOJMC2ILP6I4FA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.632835304 (376h45m35.403476649s) [noise=true , address=104.36.82.3:4791 , piece_id=NG7KRWBROBZWNP63A4734KWSRNGFVGLLXFZ5PGXZDJXFDTZNCZLQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.633120529 (376h45m35.403761864s) [noise=true , address=77.90.31.14:4692 , piece_id=NHO2352W7MMVGRS3MNPWSEFHWBOK73SQ6JQJEWZ5TV5UDFY44H5Q, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.636119667 (376h45m35.406761002s) [noise=true , address=104.36.82.2:4757 , piece_id=JMJIECO7WCFHWASHRCTEAG6VCU4S25SEETORWGJJQ63VAW533YJA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.636833744 (376h45m35.407475079s) [noise=true , address=77.90.31.15:4712 , piece_id=3DN3NPUYRN4AIBYBV2UCGE4TTYXD2WJRLHDMQEFIU36HBTYDOJ5Q, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.637112587 (376h45m35.407753922s) [noise=true , address=77.90.31.13:4643 , piece_id=PO24YVCOVVNORI7NRD3CDUC4ZPRULAXUESGBEDHJZYYDJ2ZUPQDA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:02.137197163 (376h45m36.907838498s) [upload=301417]: Next returned nil segment
Jul 9 01:56:02.137214166 (376h45m36.907855501s) [upload=301417]: Waiting for error group managing segments...
Jul 9 01:56:02.521317983 (376h45m37.291959318s) [noise=true , address=79.127.201.214:4308 , piece_id=V4LWWG3RM4JTLZGNZ3XJDHEOZO7XW5R7BHQQJ6Z4Y47PZCEZHN6A, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.567664820 (376h45m37.338306156s) [noise=true , address=77.90.31.14:4692 , piece_id=NHO2352W7MMVGRS3MNPWSEFHWBOK73SQ6JQJEWZ5TV5UDFY44H5Q, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.568760968 (376h45m37.339402303s) [noise=true , address=77.90.31.10:4539 , piece_id=OI7V6HU5WMYZSQCS5PAN6OUCAJY4UGUZVTESMLB4LHOQJVMKJLHQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.573315612 (376h45m37.343956947s) [noise=true , address=77.90.31.15:4712 , piece_id=3DN3NPUYRN4AIBYBV2UCGE4TTYXD2WJRLHDMQEFIU36HBTYDOJ5Q, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.577761728 (376h45m37.348403053s) [noise=true , address=77.90.31.19:4850 , piece_id=DNYDFQHORBVHQTPKFLWEGQF6QRLE6IIE6YGTRP6SS63CPWVKRFFA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.579676210 (376h45m37.350317545s) [noise=true , address=77.90.31.12:4615 , piece_id=GLIW3756SYGXSX44VFMMDNYVEJPS6O4ZBNS53OJRV67QNGMGLPUA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.595175322 (376h45m37.365816657s) [noise=true , address=77.90.31.16:4743 , piece_id=2VSOXKLN6Q434ET7JYLRSU6BW7PIKNZ43MZOQ4HQL6MDP3WUJOZA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.601273909 (376h45m37.371915244s) [noise=true , address=77.90.31.13:4643 , piece_id=PO24YVCOVVNORI7NRD3CDUC4ZPRULAXUESGBEDHJZYYDJ2ZUPQDA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:07.782538378 (376h45m42.553179714s) [noise=true , address=45.91.116.5:4322 , piece_id=QMOE7Z74V4HHQO7LIN6KAJHEUSDKIAVC4XLFH3AOJMC2ILP6I4FA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:22.662088184 (376h45m57.432729519s) [noise=true , address=104.36.82.17:4333 , piece_id=TPALHRSYWC3XKW77CPA36EZOOJHPLUEN4V6DWFN5NHLIAFSN5KRQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:24.053592013 (376h45m58.824233348s) [noise=true , address=104.36.82.2:4757 , piece_id=JMJIECO7WCFHWASHRCTEAG6VCU4S25SEETORWGJJQ63VAW533YJA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:44.900784468 (376h46m19.671425793s) [noise=true , address=104.36.82.23:4473 , piece_id=V3RAX7LZBU744EQQXDEG6Z25UUGQUFSSQCWCXNZKNGCTHD375GCA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:47.726372911 (376h46m22.497014236s) [noise=true , address=104.36.82.60:4364 , piece_id=7WZJU4UIO4FLPUNWUH2PVIIH3ZWWVZCP2D2UUSCNBCIS727VVWPA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:56.047624046 (376h46m30.818265381s) [noise=true , address=104.36.82.3:4791 , piece_id=NG7KRWBROBZWNP63A4734KWSRNGFVGLLXFZ5PGXZDJXFDTZNCZLQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:56.047644345 (376h46m30.81828568s) [seg_pos={0 0}, upload=301417]: Segment reached optimal threshold of 14 pieces.
Jul 9 01:56:56.047718136 (376h46m30.818359471s) [noise=true , address=104.36.82.38:4708 , piece_id=RDIWOSET3Q2JKHRQ7HVUSVMNILHVEDQ7XYDL5DMWK37Q7YTRADQA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BdJtXzvjMoNoQUjtHETyADPcZBTzcFzH8YDd651t7eTiFb1wf): protocol: expected piece hash; context canceled; EOF
Jul 9 01:56:56.047756840 (376h46m30.818398175s) [noise=true , address=104.36.82.22:4532 , piece_id=3BRLUKXPLFBXIPKURV3BBSTA562UWWQBZ75HNYXEX75E74FKJIGQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cHUuyc42zpw5k9Lz5Qo9AW9DMvKFPxRNdKZDZtjFaeyzT4Nkp): protocol: expected piece hash; context canceled; EOF
Jul 9 01:56:56.047948587 (376h46m30.818589922s) [noise=true , address=104.36.82.21:4508 , piece_id=VVX5U234EZZNCZWIVYJVYBFDYG2SCKWS5HT5GM4H7CNNJQXDHT3Q, seg_pos={0 0}, upload=301417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1269NRnqpabU6H8q9vSCmE8yMzznD1L7dyZFmzCWhtzJaTjbXg3): protocol: write tcp 188.166.225.170:60172->104.36.82.21:4508: use of closed network connection; write tcp 188.166.225.170:60172->104.36.82.21:4508: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60172->104.36.82.21:4508: use of closed network connection
Jul 9 01:56:56.047987552 (376h46m30.818628887s) [noise=true , address=104.36.82.6:4880 , piece_id=NKXAOUHYMQ4WR6V6ZGQOKRZWLAGI44AF4SNUR4OKUNJNICD4UUNQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125NsQhq7kRmTjKS8nuc51PZddApFEi5C38FzC5pEX2QrWYKzSW): protocol: write tcp 188.166.225.170:38944->104.36.82.6:4880: use of closed network connection; write tcp 188.166.225.170:38944->104.36.82.6:4880: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38944->104.36.82.6:4880: use of closed network connection
Jul 9 01:56:56.047996138 (376h46m30.818637463s) [seg_pos={0 0}, upload=301417]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 01:56:56.048046284 (376h46m30.818687619s) [upload=301417]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 01:56:56.048050091 (376h46m30.818691416s) [upload=301417]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 01:56:56.339173192 (376h46m31.109814527s) [upload=301417]: Done uploading segments...
Jul 9 01:56:56.339190416 (376h46m31.109831751s) [upload=301417]: Upload finished. err: <nil>
Duration: 50.59082409s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [......X......................X.................X......X.....] duration:
5.737662423s
FAIL (2025-07-09 02:06:13 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
nceled; EOF
Jul 9 02:00:49.060314467 (376h50m23.830955812s) [seg_pos={0 0}, upload=301481]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil>
Jul 9 02:00:49.060367628 (376h50m23.831008963s) [upload=301481]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:00:49.060371145 (376h50m23.83101248s) [upload=301481]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:00:49.357373945 (376h50m24.12801528s) [upload=301481]: Done uploading segments...
Jul 9 02:00:49.357395647 (376h50m24.128036982s) [upload=301481]: Upload finished. err: <nil>
Jul 9 02:02:01.314289870 (376h51m36.084931205s) [upload=301495]: Starting upload
Jul 9 02:02:01.314394990 (376h51m36.085036325s) [upload=301495]: Done starting upload
Jul 9 02:02:01.314484112 (376h51m36.085125457s) [upload=301495]: Uploading segments...
Jul 9 02:02:01.314699112 (376h51m36.085340457s) [upload=301495]: Got next segment. Inline: false
Jul 9 02:02:01.314747194 (376h51m36.085388529s) [upload=301495]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:02:01.314753927 (376h51m36.085395272s) [upload=301495]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:02:01.549200488 (376h51m36.319841873s) [seg_pos={0 0}, upload=301495]: Begin upload segment...
Jul 9 02:02:01.549266264 (376h51m36.319907609s) [seg_pos={0 0}, upload=301495]: Done begin upload segment.
Jul 9 02:02:01.549310128 (376h51m36.319951473s) [noise=true , address=104.36.82.7:4910 , piece_id=CPROZR2V6J7RKRZW6CNLA7P44DQVYH3VOZMYBBTEIBOL7MGINPDA, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.549336468 (376h51m36.319977823s) [noise=true , address=45.91.116.14:4190 , piece_id=ZAF4GOHABQQENBFRPE3HEMFTT474ULDEB5MZ5XDVVHFVTQID5AVA, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.549964509 (376h51m36.320605844s) [noise=true , address=104.36.82.13:4187 , piece_id=BLI6GHKTU3D5Q6FWTOS6UEPDLBDNCX2VEZCKAXIVAF32VWU7I6XQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.550253653 (376h51m36.320894988s) [noise=true , address=104.36.82.9:4006 , piece_id=XX7OUX5VBH5UVHELOP525HLHU5LA7O4B6DR6LKI7VYFWZYVAXHTQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.550518870 (376h51m36.321160225s) [noise=true , address=104.36.82.6:4874 , piece_id=UDKHL2V3C6ZYBR3FGHVIKQYJLPCOO3XLXYTSQJVLTGJ6I74IGGAA, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.550556472 (376h51m36.321197807s) [noise=true , address=45.91.116.10:4082 , piece_id=IRWOX4OQI5TB7FAT53RSX4C6DPNPUGZPPQXS6VFM2EE6FPZI4N4A, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.550837249 (376h51m36.321478584s) [noise=true , address=45.91.116.8:4048 , piece_id=UTXJQHFKWTANYVEEL2CICL5MQWVLUC4H4ZQ3IFXQD5O45RKB425A, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.551078060 (376h51m36.321719395s) [noise=true , address=45.91.116.7:4010 , piece_id=7CYJB5QPC3ZJZVGFNJKBZHOLSKPFBYMSWTJL37CQ73JIIM7LVA2A, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.551313831 (376h51m36.321955166s) [noise=true , address=77.90.31.11:4555 , piece_id=57OEP4BSHFGZQ5A7DJKJ4BNSO7ORPXPVMSN2I56YFREXE6N5RIPQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.551369758 (376h51m36.322011093s) [noise=true , address=45.91.116.11:4119 , piece_id=2TR3FC2ZYYNAPWL2GTL5EZVJW74NI5W275AYDNSZIME62TPILDBQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.551601652 (376h51m36.322242987s) [noise=true , address=104.36.82.5:4839 , piece_id=MM6CHXUZFEON5CG7GUKRLD7PGYKU3337UBU73DTHSKAKQUJUKYLQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.551803969 (376h51m36.322445304s) [noise=true , address=45.91.116.6:4973 , piece_id=2XDTRFYDWADB2U2DIT6VDWTHT6DVRPRNMCWJFW4Q7G7KXITLI6LQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.552051763 (376h51m36.322693098s) [noise=true , address=104.36.82.15:4258 , piece_id=KFL7XC6ILPLWJGYIGZVJ6X3ECS26XNNY6M4ZS3QHGXBPZG4XWILQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.552343200 (376h51m36.322984545s) [noise=true , address=45.91.116.13:4154 , piece_id=I2XLUYF36R6HKCEYHRUISRDMAPYP7CQHEXL7XB6IMG7GYO32W55Q, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.552474742 (376h51m36.323116077s) [noise=true , address=45.91.116.4:4939 , piece_id=QDXP222CYJXWF2NV5LKI7XOPMU7APMWDFQLWURFQG5BVJUNKNJBA, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.552718097 (376h51m36.323359433s) [noise=true , address=104.36.82.11:4114 , piece_id=R7MKGZKIKESIOB5CWCDO4KV2CNGN2EXTLGATSMK7Y2JMSFIB3DSA, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:03.251350348 (376h51m38.021991683s) [upload=301495]: Next returned nil segment
Jul 9 02:02:03.251367120 (376h51m38.022008455s) [upload=301495]: Waiting for error group managing segments...
Jul 9 02:02:03.536295419 (376h51m38.306936775s) [noise=true , address=104.36.82.9:4006 , piece_id=XX7OUX5VBH5UVHELOP525HLHU5LA7O4B6DR6LKI7VYFWZYVAXHTQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.585972469 (376h51m38.356613794s) [noise=true , address=45.91.116.8:4048 , piece_id=UTXJQHFKWTANYVEEL2CICL5MQWVLUC4H4ZQ3IFXQD5O45RKB425A, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.612355463 (376h51m38.382996798s) [noise=true , address=45.91.116.11:4119 , piece_id=2TR3FC2ZYYNAPWL2GTL5EZVJW74NI5W275AYDNSZIME62TPILDBQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.616140875 (376h51m38.38678221s) [noise=true , address=45.91.116.4:4939 , piece_id=QDXP222CYJXWF2NV5LKI7XOPMU7APMWDFQLWURFQG5BVJUNKNJBA, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.625063725 (376h51m38.395705061s) [noise=true , address=45.91.116.13:4154 , piece_id=I2XLUYF36R6HKCEYHRUISRDMAPYP7CQHEXL7XB6IMG7GYO32W55Q, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.627327436 (376h51m38.397968771s) [noise=true , address=77.90.31.11:4555 , piece_id=57OEP4BSHFGZQ5A7DJKJ4BNSO7ORPXPVMSN2I56YFREXE6N5RIPQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.680079459 (376h51m38.450720804s) [noise=true , address=45.91.116.14:4190 , piece_id=ZAF4GOHABQQENBFRPE3HEMFTT474ULDEB5MZ5XDVVHFVTQID5AVA, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.682275680 (376h51m38.452917015s) [noise=true , address=45.91.116.10:4082 , piece_id=IRWOX4OQI5TB7FAT53RSX4C6DPNPUGZPPQXS6VFM2EE6FPZI4N4A, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.693418348 (376h51m38.464059684s) [noise=true , address=45.91.116.6:4973 , piece_id=2XDTRFYDWADB2U2DIT6VDWTHT6DVRPRNMCWJFW4Q7G7KXITLI6LQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.714112589 (376h51m38.484753924s) [noise=true , address=45.91.116.7:4010 , piece_id=7CYJB5QPC3ZJZVGFNJKBZHOLSKPFBYMSWTJL37CQ73JIIM7LVA2A, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:24.743005877 (376h51m59.513647202s) [noise=true , address=104.36.82.6:4874 , piece_id=UDKHL2V3C6ZYBR3FGHVIKQYJLPCOO3XLXYTSQJVLTGJ6I74IGGAA, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:28.578275204 (376h52m3.348916539s) [noise=true , address=104.36.82.13:4187 , piece_id=BLI6GHKTU3D5Q6FWTOS6UEPDLBDNCX2VEZCKAXIVAF32VWU7I6XQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:30.664723094 (376h52m5.435364439s) [noise=true , address=104.36.82.11:4114 , piece_id=R7MKGZKIKESIOB5CWCDO4KV2CNGN2EXTLGATSMK7Y2JMSFIB3DSA, seg_pos={0 0}, upload=301495]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:02:30.665014852 (376h52m5.435656188s) [noise=true , address=104.36.82.5:4839 , piece_id=MM6CHXUZFEON5CG7GUKRLD7PGYKU3337UBU73DTHSKAKQUJUKYLQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 188.166.225.170:42438->104.36.82.5:4839: use of closed network connection; write tcp 188.166.225.170:42438->104.36.82.5:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42438->104.36.82.5:4839: use of closed network connection
Jul 9 02:02:30.665113180 (376h52m5.435754525s) [noise=true , address=104.36.82.7:4910 , piece_id=CPROZR2V6J7RKRZW6CNLA7P44DQVYH3VOZMYBBTEIBOL7MGINPDA, seg_pos={0 0}, upload=301495]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: write tcp 188.166.225.170:38368->104.36.82.7:4910: use of closed network connection; write tcp 188.166.225.170:38368->104.36.82.7:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38368->104.36.82.7:4910: use of closed network connection
Jul 9 02:02:30.665314215 (376h52m5.43595555s) [noise=true , address=104.36.82.15:4258 , piece_id=KFL7XC6ILPLWJGYIGZVJ6X3ECS26XNNY6M4ZS3QHGXBPZG4XWILQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 188.166.225.170:50202->104.36.82.15:4258: use of closed network connection; context deadline exceeded
Jul 9 02:02:30.665353560 (376h52m5.435994895s) [seg_pos={0 0}, upload=301495]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 02:02:30.900452118 (376h52m5.671093453s) [upload=301495]: Done uploading segments...
Jul 9 02:02:30.900486934 (376h52m5.67112827s) [upload=301495]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.737702219s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [X.X..X...X..X............X......X..X...X.X.X.X...X..........] duration:
14.158687001s
FAIL (2025-07-09 02:06:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 14.15870228s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.......X......................X.................X......X....] duration:
23.001020209s
FAIL (2025-07-09 02:06:04 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
nceled; EOF
Jul 9 02:00:49.060314467 (376h50m23.830955812s) [seg_pos={0 0}, upload=301481]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil>
Jul 9 02:00:49.060367628 (376h50m23.831008963s) [upload=301481]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:00:49.060371145 (376h50m23.83101248s) [upload=301481]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:00:49.357373945 (376h50m24.12801528s) [upload=301481]: Done uploading segments...
Jul 9 02:00:49.357395647 (376h50m24.128036982s) [upload=301481]: Upload finished. err: <nil>
Jul 9 02:02:01.314289870 (376h51m36.084931205s) [upload=301495]: Starting upload
Jul 9 02:02:01.314394990 (376h51m36.085036325s) [upload=301495]: Done starting upload
Jul 9 02:02:01.314484112 (376h51m36.085125457s) [upload=301495]: Uploading segments...
Jul 9 02:02:01.314699112 (376h51m36.085340457s) [upload=301495]: Got next segment. Inline: false
Jul 9 02:02:01.314747194 (376h51m36.085388529s) [upload=301495]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:02:01.314753927 (376h51m36.085395272s) [upload=301495]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:02:01.549200488 (376h51m36.319841873s) [seg_pos={0 0}, upload=301495]: Begin upload segment...
Jul 9 02:02:01.549266264 (376h51m36.319907609s) [seg_pos={0 0}, upload=301495]: Done begin upload segment.
Jul 9 02:02:01.549310128 (376h51m36.319951473s) [noise=true , address=104.36.82.7:4910 , piece_id=CPROZR2V6J7RKRZW6CNLA7P44DQVYH3VOZMYBBTEIBOL7MGINPDA, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.549336468 (376h51m36.319977823s) [noise=true , address=45.91.116.14:4190 , piece_id=ZAF4GOHABQQENBFRPE3HEMFTT474ULDEB5MZ5XDVVHFVTQID5AVA, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.549964509 (376h51m36.320605844s) [noise=true , address=104.36.82.13:4187 , piece_id=BLI6GHKTU3D5Q6FWTOS6UEPDLBDNCX2VEZCKAXIVAF32VWU7I6XQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.550253653 (376h51m36.320894988s) [noise=true , address=104.36.82.9:4006 , piece_id=XX7OUX5VBH5UVHELOP525HLHU5LA7O4B6DR6LKI7VYFWZYVAXHTQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.550518870 (376h51m36.321160225s) [noise=true , address=104.36.82.6:4874 , piece_id=UDKHL2V3C6ZYBR3FGHVIKQYJLPCOO3XLXYTSQJVLTGJ6I74IGGAA, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.550556472 (376h51m36.321197807s) [noise=true , address=45.91.116.10:4082 , piece_id=IRWOX4OQI5TB7FAT53RSX4C6DPNPUGZPPQXS6VFM2EE6FPZI4N4A, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.550837249 (376h51m36.321478584s) [noise=true , address=45.91.116.8:4048 , piece_id=UTXJQHFKWTANYVEEL2CICL5MQWVLUC4H4ZQ3IFXQD5O45RKB425A, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.551078060 (376h51m36.321719395s) [noise=true , address=45.91.116.7:4010 , piece_id=7CYJB5QPC3ZJZVGFNJKBZHOLSKPFBYMSWTJL37CQ73JIIM7LVA2A, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.551313831 (376h51m36.321955166s) [noise=true , address=77.90.31.11:4555 , piece_id=57OEP4BSHFGZQ5A7DJKJ4BNSO7ORPXPVMSN2I56YFREXE6N5RIPQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.551369758 (376h51m36.322011093s) [noise=true , address=45.91.116.11:4119 , piece_id=2TR3FC2ZYYNAPWL2GTL5EZVJW74NI5W275AYDNSZIME62TPILDBQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.551601652 (376h51m36.322242987s) [noise=true , address=104.36.82.5:4839 , piece_id=MM6CHXUZFEON5CG7GUKRLD7PGYKU3337UBU73DTHSKAKQUJUKYLQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.551803969 (376h51m36.322445304s) [noise=true , address=45.91.116.6:4973 , piece_id=2XDTRFYDWADB2U2DIT6VDWTHT6DVRPRNMCWJFW4Q7G7KXITLI6LQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.552051763 (376h51m36.322693098s) [noise=true , address=104.36.82.15:4258 , piece_id=KFL7XC6ILPLWJGYIGZVJ6X3ECS26XNNY6M4ZS3QHGXBPZG4XWILQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.552343200 (376h51m36.322984545s) [noise=true , address=45.91.116.13:4154 , piece_id=I2XLUYF36R6HKCEYHRUISRDMAPYP7CQHEXL7XB6IMG7GYO32W55Q, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.552474742 (376h51m36.323116077s) [noise=true , address=45.91.116.4:4939 , piece_id=QDXP222CYJXWF2NV5LKI7XOPMU7APMWDFQLWURFQG5BVJUNKNJBA, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.552718097 (376h51m36.323359433s) [noise=true , address=104.36.82.11:4114 , piece_id=R7MKGZKIKESIOB5CWCDO4KV2CNGN2EXTLGATSMK7Y2JMSFIB3DSA, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:03.251350348 (376h51m38.021991683s) [upload=301495]: Next returned nil segment
Jul 9 02:02:03.251367120 (376h51m38.022008455s) [upload=301495]: Waiting for error group managing segments...
Jul 9 02:02:03.536295419 (376h51m38.306936775s) [noise=true , address=104.36.82.9:4006 , piece_id=XX7OUX5VBH5UVHELOP525HLHU5LA7O4B6DR6LKI7VYFWZYVAXHTQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.585972469 (376h51m38.356613794s) [noise=true , address=45.91.116.8:4048 , piece_id=UTXJQHFKWTANYVEEL2CICL5MQWVLUC4H4ZQ3IFXQD5O45RKB425A, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.612355463 (376h51m38.382996798s) [noise=true , address=45.91.116.11:4119 , piece_id=2TR3FC2ZYYNAPWL2GTL5EZVJW74NI5W275AYDNSZIME62TPILDBQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.616140875 (376h51m38.38678221s) [noise=true , address=45.91.116.4:4939 , piece_id=QDXP222CYJXWF2NV5LKI7XOPMU7APMWDFQLWURFQG5BVJUNKNJBA, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.625063725 (376h51m38.395705061s) [noise=true , address=45.91.116.13:4154 , piece_id=I2XLUYF36R6HKCEYHRUISRDMAPYP7CQHEXL7XB6IMG7GYO32W55Q, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.627327436 (376h51m38.397968771s) [noise=true , address=77.90.31.11:4555 , piece_id=57OEP4BSHFGZQ5A7DJKJ4BNSO7ORPXPVMSN2I56YFREXE6N5RIPQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.680079459 (376h51m38.450720804s) [noise=true , address=45.91.116.14:4190 , piece_id=ZAF4GOHABQQENBFRPE3HEMFTT474ULDEB5MZ5XDVVHFVTQID5AVA, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.682275680 (376h51m38.452917015s) [noise=true , address=45.91.116.10:4082 , piece_id=IRWOX4OQI5TB7FAT53RSX4C6DPNPUGZPPQXS6VFM2EE6FPZI4N4A, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.693418348 (376h51m38.464059684s) [noise=true , address=45.91.116.6:4973 , piece_id=2XDTRFYDWADB2U2DIT6VDWTHT6DVRPRNMCWJFW4Q7G7KXITLI6LQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.714112589 (376h51m38.484753924s) [noise=true , address=45.91.116.7:4010 , piece_id=7CYJB5QPC3ZJZVGFNJKBZHOLSKPFBYMSWTJL37CQ73JIIM7LVA2A, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:24.743005877 (376h51m59.513647202s) [noise=true , address=104.36.82.6:4874 , piece_id=UDKHL2V3C6ZYBR3FGHVIKQYJLPCOO3XLXYTSQJVLTGJ6I74IGGAA, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:28.578275204 (376h52m3.348916539s) [noise=true , address=104.36.82.13:4187 , piece_id=BLI6GHKTU3D5Q6FWTOS6UEPDLBDNCX2VEZCKAXIVAF32VWU7I6XQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:30.664723094 (376h52m5.435364439s) [noise=true , address=104.36.82.11:4114 , piece_id=R7MKGZKIKESIOB5CWCDO4KV2CNGN2EXTLGATSMK7Y2JMSFIB3DSA, seg_pos={0 0}, upload=301495]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:02:30.665014852 (376h52m5.435656188s) [noise=true , address=104.36.82.5:4839 , piece_id=MM6CHXUZFEON5CG7GUKRLD7PGYKU3337UBU73DTHSKAKQUJUKYLQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 188.166.225.170:42438->104.36.82.5:4839: use of closed network connection; write tcp 188.166.225.170:42438->104.36.82.5:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42438->104.36.82.5:4839: use of closed network connection
Jul 9 02:02:30.665113180 (376h52m5.435754525s) [noise=true , address=104.36.82.7:4910 , piece_id=CPROZR2V6J7RKRZW6CNLA7P44DQVYH3VOZMYBBTEIBOL7MGINPDA, seg_pos={0 0}, upload=301495]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: write tcp 188.166.225.170:38368->104.36.82.7:4910: use of closed network connection; write tcp 188.166.225.170:38368->104.36.82.7:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38368->104.36.82.7:4910: use of closed network connection
Jul 9 02:02:30.665314215 (376h52m5.43595555s) [noise=true , address=104.36.82.15:4258 , piece_id=KFL7XC6ILPLWJGYIGZVJ6X3ECS26XNNY6M4ZS3QHGXBPZG4XWILQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 188.166.225.170:50202->104.36.82.15:4258: use of closed network connection; context deadline exceeded
Jul 9 02:02:30.665353560 (376h52m5.435994895s) [seg_pos={0 0}, upload=301495]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 02:02:30.900452118 (376h52m5.671093453s) [upload=301495]: Done uploading segments...
Jul 9 02:02:30.900486934 (376h52m5.67112827s) [upload=301495]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 23.001046701s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
18 out of 60 is failed (threshold=3) [.....X......XX.XX..X......XXX....XX.....XX...X.XX........X.X] duration:
1m0.001204517s
FAIL (2025-07-09 02:05:34 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.001225207s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [...............X......X..........X.............X............] duration:
41.469925971s
FAIL (2025-07-09 02:05:29 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
hRequestItem_SegmentBegin
Jul 9 01:56:00.629444016 (376h45m35.400085361s) [seg_pos={0 0}, upload=301417]: Begin upload segment...
Jul 9 01:56:00.629512196 (376h45m35.400153531s) [seg_pos={0 0}, upload=301417]: Done begin upload segment.
Jul 9 01:56:00.629572552 (376h45m35.400213887s) [noise=true , address=104.36.82.22:4532 , piece_id=3BRLUKXPLFBXIPKURV3BBSTA562UWWQBZ75HNYXEX75E74FKJIGQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.629579284 (376h45m35.400220619s) [noise=true , address=77.90.31.10:4539 , piece_id=OI7V6HU5WMYZSQCS5PAN6OUCAJY4UGUZVTESMLB4LHOQJVMKJLHQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.629690497 (376h45m35.400331832s) [noise=true , address=104.36.82.17:4333 , piece_id=TPALHRSYWC3XKW77CPA36EZOOJHPLUEN4V6DWFN5NHLIAFSN5KRQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630082818 (376h45m35.400724153s) [noise=true , address=104.36.82.6:4880 , piece_id=NKXAOUHYMQ4WR6V6ZGQOKRZWLAGI44AF4SNUR4OKUNJNICD4UUNQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630158242 (376h45m35.400799577s) [noise=true , address=79.127.201.214:4308 , piece_id=V4LWWG3RM4JTLZGNZ3XJDHEOZO7XW5R7BHQQJ6Z4Y47PZCEZHN6A, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630428680 (376h45m35.401070025s) [noise=true , address=77.90.31.12:4615 , piece_id=GLIW3756SYGXSX44VFMMDNYVEJPS6O4ZBNS53OJRV67QNGMGLPUA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630538129 (376h45m35.401179464s) [noise=true , address=104.36.82.60:4364 , piece_id=7WZJU4UIO4FLPUNWUH2PVIIH3ZWWVZCP2D2UUSCNBCIS727VVWPA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630862820 (376h45m35.401504165s) [noise=true , address=104.36.82.38:4708 , piece_id=RDIWOSET3Q2JKHRQ7HVUSVMNILHVEDQ7XYDL5DMWK37Q7YTRADQA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.631210084 (376h45m35.401851419s) [noise=true , address=104.36.82.21:4508 , piece_id=VVX5U234EZZNCZWIVYJVYBFDYG2SCKWS5HT5GM4H7CNNJQXDHT3Q, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.631555495 (376h45m35.40219684s) [noise=true , address=77.90.31.16:4743 , piece_id=2VSOXKLN6Q434ET7JYLRSU6BW7PIKNZ43MZOQ4HQL6MDP3WUJOZA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.631989165 (376h45m35.4026305s) [noise=true , address=104.36.82.23:4473 , piece_id=V3RAX7LZBU744EQQXDEG6Z25UUGQUFSSQCWCXNZKNGCTHD375GCA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.632258450 (376h45m35.402899785s) [noise=true , address=77.90.31.19:4850 , piece_id=DNYDFQHORBVHQTPKFLWEGQF6QRLE6IIE6YGTRP6SS63CPWVKRFFA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.632521984 (376h45m35.403163309s) [noise=true , address=45.91.116.5:4322 , piece_id=QMOE7Z74V4HHQO7LIN6KAJHEUSDKIAVC4XLFH3AOJMC2ILP6I4FA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.632835304 (376h45m35.403476649s) [noise=true , address=104.36.82.3:4791 , piece_id=NG7KRWBROBZWNP63A4734KWSRNGFVGLLXFZ5PGXZDJXFDTZNCZLQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.633120529 (376h45m35.403761864s) [noise=true , address=77.90.31.14:4692 , piece_id=NHO2352W7MMVGRS3MNPWSEFHWBOK73SQ6JQJEWZ5TV5UDFY44H5Q, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.636119667 (376h45m35.406761002s) [noise=true , address=104.36.82.2:4757 , piece_id=JMJIECO7WCFHWASHRCTEAG6VCU4S25SEETORWGJJQ63VAW533YJA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.636833744 (376h45m35.407475079s) [noise=true , address=77.90.31.15:4712 , piece_id=3DN3NPUYRN4AIBYBV2UCGE4TTYXD2WJRLHDMQEFIU36HBTYDOJ5Q, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.637112587 (376h45m35.407753922s) [noise=true , address=77.90.31.13:4643 , piece_id=PO24YVCOVVNORI7NRD3CDUC4ZPRULAXUESGBEDHJZYYDJ2ZUPQDA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:02.137197163 (376h45m36.907838498s) [upload=301417]: Next returned nil segment
Jul 9 01:56:02.137214166 (376h45m36.907855501s) [upload=301417]: Waiting for error group managing segments...
Jul 9 01:56:02.521317983 (376h45m37.291959318s) [noise=true , address=79.127.201.214:4308 , piece_id=V4LWWG3RM4JTLZGNZ3XJDHEOZO7XW5R7BHQQJ6Z4Y47PZCEZHN6A, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.567664820 (376h45m37.338306156s) [noise=true , address=77.90.31.14:4692 , piece_id=NHO2352W7MMVGRS3MNPWSEFHWBOK73SQ6JQJEWZ5TV5UDFY44H5Q, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.568760968 (376h45m37.339402303s) [noise=true , address=77.90.31.10:4539 , piece_id=OI7V6HU5WMYZSQCS5PAN6OUCAJY4UGUZVTESMLB4LHOQJVMKJLHQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.573315612 (376h45m37.343956947s) [noise=true , address=77.90.31.15:4712 , piece_id=3DN3NPUYRN4AIBYBV2UCGE4TTYXD2WJRLHDMQEFIU36HBTYDOJ5Q, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.577761728 (376h45m37.348403053s) [noise=true , address=77.90.31.19:4850 , piece_id=DNYDFQHORBVHQTPKFLWEGQF6QRLE6IIE6YGTRP6SS63CPWVKRFFA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.579676210 (376h45m37.350317545s) [noise=true , address=77.90.31.12:4615 , piece_id=GLIW3756SYGXSX44VFMMDNYVEJPS6O4ZBNS53OJRV67QNGMGLPUA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.595175322 (376h45m37.365816657s) [noise=true , address=77.90.31.16:4743 , piece_id=2VSOXKLN6Q434ET7JYLRSU6BW7PIKNZ43MZOQ4HQL6MDP3WUJOZA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.601273909 (376h45m37.371915244s) [noise=true , address=77.90.31.13:4643 , piece_id=PO24YVCOVVNORI7NRD3CDUC4ZPRULAXUESGBEDHJZYYDJ2ZUPQDA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:07.782538378 (376h45m42.553179714s) [noise=true , address=45.91.116.5:4322 , piece_id=QMOE7Z74V4HHQO7LIN6KAJHEUSDKIAVC4XLFH3AOJMC2ILP6I4FA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:22.662088184 (376h45m57.432729519s) [noise=true , address=104.36.82.17:4333 , piece_id=TPALHRSYWC3XKW77CPA36EZOOJHPLUEN4V6DWFN5NHLIAFSN5KRQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:24.053592013 (376h45m58.824233348s) [noise=true , address=104.36.82.2:4757 , piece_id=JMJIECO7WCFHWASHRCTEAG6VCU4S25SEETORWGJJQ63VAW533YJA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:44.900784468 (376h46m19.671425793s) [noise=true , address=104.36.82.23:4473 , piece_id=V3RAX7LZBU744EQQXDEG6Z25UUGQUFSSQCWCXNZKNGCTHD375GCA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:47.726372911 (376h46m22.497014236s) [noise=true , address=104.36.82.60:4364 , piece_id=7WZJU4UIO4FLPUNWUH2PVIIH3ZWWVZCP2D2UUSCNBCIS727VVWPA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:56.047624046 (376h46m30.818265381s) [noise=true , address=104.36.82.3:4791 , piece_id=NG7KRWBROBZWNP63A4734KWSRNGFVGLLXFZ5PGXZDJXFDTZNCZLQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:56.047644345 (376h46m30.81828568s) [seg_pos={0 0}, upload=301417]: Segment reached optimal threshold of 14 pieces.
Jul 9 01:56:56.047718136 (376h46m30.818359471s) [noise=true , address=104.36.82.38:4708 , piece_id=RDIWOSET3Q2JKHRQ7HVUSVMNILHVEDQ7XYDL5DMWK37Q7YTRADQA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BdJtXzvjMoNoQUjtHETyADPcZBTzcFzH8YDd651t7eTiFb1wf): protocol: expected piece hash; context canceled; EOF
Jul 9 01:56:56.047756840 (376h46m30.818398175s) [noise=true , address=104.36.82.22:4532 , piece_id=3BRLUKXPLFBXIPKURV3BBSTA562UWWQBZ75HNYXEX75E74FKJIGQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cHUuyc42zpw5k9Lz5Qo9AW9DMvKFPxRNdKZDZtjFaeyzT4Nkp): protocol: expected piece hash; context canceled; EOF
Jul 9 01:56:56.047948587 (376h46m30.818589922s) [noise=true , address=104.36.82.21:4508 , piece_id=VVX5U234EZZNCZWIVYJVYBFDYG2SCKWS5HT5GM4H7CNNJQXDHT3Q, seg_pos={0 0}, upload=301417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1269NRnqpabU6H8q9vSCmE8yMzznD1L7dyZFmzCWhtzJaTjbXg3): protocol: write tcp 188.166.225.170:60172->104.36.82.21:4508: use of closed network connection; write tcp 188.166.225.170:60172->104.36.82.21:4508: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60172->104.36.82.21:4508: use of closed network connection
Jul 9 01:56:56.047987552 (376h46m30.818628887s) [noise=true , address=104.36.82.6:4880 , piece_id=NKXAOUHYMQ4WR6V6ZGQOKRZWLAGI44AF4SNUR4OKUNJNICD4UUNQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125NsQhq7kRmTjKS8nuc51PZddApFEi5C38FzC5pEX2QrWYKzSW): protocol: write tcp 188.166.225.170:38944->104.36.82.6:4880: use of closed network connection; write tcp 188.166.225.170:38944->104.36.82.6:4880: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38944->104.36.82.6:4880: use of closed network connection
Jul 9 01:56:56.047996138 (376h46m30.818637463s) [seg_pos={0 0}, upload=301417]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 01:56:56.048046284 (376h46m30.818687619s) [upload=301417]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 01:56:56.048050091 (376h46m30.818691416s) [upload=301417]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 01:56:56.339173192 (376h46m31.109814527s) [upload=301417]: Done uploading segments...
Jul 9 01:56:56.339190416 (376h46m31.109831751s) [upload=301417]: Upload finished. err: <nil>
Duration: 41.469941661s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [.X.X..X...X..X............X......X..X...X.X.X.X...X.........] duration:
5.822445407s
FAIL (2025-07-09 02:04:59 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.822465525s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [................X......X..........X.............X...........] duration:
40.455789209s
FAIL (2025-07-09 02:04:47 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
hRequestItem_SegmentBegin
Jul 9 01:56:00.629444016 (376h45m35.400085361s) [seg_pos={0 0}, upload=301417]: Begin upload segment...
Jul 9 01:56:00.629512196 (376h45m35.400153531s) [seg_pos={0 0}, upload=301417]: Done begin upload segment.
Jul 9 01:56:00.629572552 (376h45m35.400213887s) [noise=true , address=104.36.82.22:4532 , piece_id=3BRLUKXPLFBXIPKURV3BBSTA562UWWQBZ75HNYXEX75E74FKJIGQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.629579284 (376h45m35.400220619s) [noise=true , address=77.90.31.10:4539 , piece_id=OI7V6HU5WMYZSQCS5PAN6OUCAJY4UGUZVTESMLB4LHOQJVMKJLHQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.629690497 (376h45m35.400331832s) [noise=true , address=104.36.82.17:4333 , piece_id=TPALHRSYWC3XKW77CPA36EZOOJHPLUEN4V6DWFN5NHLIAFSN5KRQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630082818 (376h45m35.400724153s) [noise=true , address=104.36.82.6:4880 , piece_id=NKXAOUHYMQ4WR6V6ZGQOKRZWLAGI44AF4SNUR4OKUNJNICD4UUNQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630158242 (376h45m35.400799577s) [noise=true , address=79.127.201.214:4308 , piece_id=V4LWWG3RM4JTLZGNZ3XJDHEOZO7XW5R7BHQQJ6Z4Y47PZCEZHN6A, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630428680 (376h45m35.401070025s) [noise=true , address=77.90.31.12:4615 , piece_id=GLIW3756SYGXSX44VFMMDNYVEJPS6O4ZBNS53OJRV67QNGMGLPUA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630538129 (376h45m35.401179464s) [noise=true , address=104.36.82.60:4364 , piece_id=7WZJU4UIO4FLPUNWUH2PVIIH3ZWWVZCP2D2UUSCNBCIS727VVWPA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630862820 (376h45m35.401504165s) [noise=true , address=104.36.82.38:4708 , piece_id=RDIWOSET3Q2JKHRQ7HVUSVMNILHVEDQ7XYDL5DMWK37Q7YTRADQA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.631210084 (376h45m35.401851419s) [noise=true , address=104.36.82.21:4508 , piece_id=VVX5U234EZZNCZWIVYJVYBFDYG2SCKWS5HT5GM4H7CNNJQXDHT3Q, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.631555495 (376h45m35.40219684s) [noise=true , address=77.90.31.16:4743 , piece_id=2VSOXKLN6Q434ET7JYLRSU6BW7PIKNZ43MZOQ4HQL6MDP3WUJOZA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.631989165 (376h45m35.4026305s) [noise=true , address=104.36.82.23:4473 , piece_id=V3RAX7LZBU744EQQXDEG6Z25UUGQUFSSQCWCXNZKNGCTHD375GCA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.632258450 (376h45m35.402899785s) [noise=true , address=77.90.31.19:4850 , piece_id=DNYDFQHORBVHQTPKFLWEGQF6QRLE6IIE6YGTRP6SS63CPWVKRFFA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.632521984 (376h45m35.403163309s) [noise=true , address=45.91.116.5:4322 , piece_id=QMOE7Z74V4HHQO7LIN6KAJHEUSDKIAVC4XLFH3AOJMC2ILP6I4FA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.632835304 (376h45m35.403476649s) [noise=true , address=104.36.82.3:4791 , piece_id=NG7KRWBROBZWNP63A4734KWSRNGFVGLLXFZ5PGXZDJXFDTZNCZLQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.633120529 (376h45m35.403761864s) [noise=true , address=77.90.31.14:4692 , piece_id=NHO2352W7MMVGRS3MNPWSEFHWBOK73SQ6JQJEWZ5TV5UDFY44H5Q, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.636119667 (376h45m35.406761002s) [noise=true , address=104.36.82.2:4757 , piece_id=JMJIECO7WCFHWASHRCTEAG6VCU4S25SEETORWGJJQ63VAW533YJA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.636833744 (376h45m35.407475079s) [noise=true , address=77.90.31.15:4712 , piece_id=3DN3NPUYRN4AIBYBV2UCGE4TTYXD2WJRLHDMQEFIU36HBTYDOJ5Q, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.637112587 (376h45m35.407753922s) [noise=true , address=77.90.31.13:4643 , piece_id=PO24YVCOVVNORI7NRD3CDUC4ZPRULAXUESGBEDHJZYYDJ2ZUPQDA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:02.137197163 (376h45m36.907838498s) [upload=301417]: Next returned nil segment
Jul 9 01:56:02.137214166 (376h45m36.907855501s) [upload=301417]: Waiting for error group managing segments...
Jul 9 01:56:02.521317983 (376h45m37.291959318s) [noise=true , address=79.127.201.214:4308 , piece_id=V4LWWG3RM4JTLZGNZ3XJDHEOZO7XW5R7BHQQJ6Z4Y47PZCEZHN6A, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.567664820 (376h45m37.338306156s) [noise=true , address=77.90.31.14:4692 , piece_id=NHO2352W7MMVGRS3MNPWSEFHWBOK73SQ6JQJEWZ5TV5UDFY44H5Q, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.568760968 (376h45m37.339402303s) [noise=true , address=77.90.31.10:4539 , piece_id=OI7V6HU5WMYZSQCS5PAN6OUCAJY4UGUZVTESMLB4LHOQJVMKJLHQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.573315612 (376h45m37.343956947s) [noise=true , address=77.90.31.15:4712 , piece_id=3DN3NPUYRN4AIBYBV2UCGE4TTYXD2WJRLHDMQEFIU36HBTYDOJ5Q, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.577761728 (376h45m37.348403053s) [noise=true , address=77.90.31.19:4850 , piece_id=DNYDFQHORBVHQTPKFLWEGQF6QRLE6IIE6YGTRP6SS63CPWVKRFFA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.579676210 (376h45m37.350317545s) [noise=true , address=77.90.31.12:4615 , piece_id=GLIW3756SYGXSX44VFMMDNYVEJPS6O4ZBNS53OJRV67QNGMGLPUA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.595175322 (376h45m37.365816657s) [noise=true , address=77.90.31.16:4743 , piece_id=2VSOXKLN6Q434ET7JYLRSU6BW7PIKNZ43MZOQ4HQL6MDP3WUJOZA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.601273909 (376h45m37.371915244s) [noise=true , address=77.90.31.13:4643 , piece_id=PO24YVCOVVNORI7NRD3CDUC4ZPRULAXUESGBEDHJZYYDJ2ZUPQDA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:07.782538378 (376h45m42.553179714s) [noise=true , address=45.91.116.5:4322 , piece_id=QMOE7Z74V4HHQO7LIN6KAJHEUSDKIAVC4XLFH3AOJMC2ILP6I4FA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:22.662088184 (376h45m57.432729519s) [noise=true , address=104.36.82.17:4333 , piece_id=TPALHRSYWC3XKW77CPA36EZOOJHPLUEN4V6DWFN5NHLIAFSN5KRQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:24.053592013 (376h45m58.824233348s) [noise=true , address=104.36.82.2:4757 , piece_id=JMJIECO7WCFHWASHRCTEAG6VCU4S25SEETORWGJJQ63VAW533YJA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:44.900784468 (376h46m19.671425793s) [noise=true , address=104.36.82.23:4473 , piece_id=V3RAX7LZBU744EQQXDEG6Z25UUGQUFSSQCWCXNZKNGCTHD375GCA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:47.726372911 (376h46m22.497014236s) [noise=true , address=104.36.82.60:4364 , piece_id=7WZJU4UIO4FLPUNWUH2PVIIH3ZWWVZCP2D2UUSCNBCIS727VVWPA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:56.047624046 (376h46m30.818265381s) [noise=true , address=104.36.82.3:4791 , piece_id=NG7KRWBROBZWNP63A4734KWSRNGFVGLLXFZ5PGXZDJXFDTZNCZLQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:56.047644345 (376h46m30.81828568s) [seg_pos={0 0}, upload=301417]: Segment reached optimal threshold of 14 pieces.
Jul 9 01:56:56.047718136 (376h46m30.818359471s) [noise=true , address=104.36.82.38:4708 , piece_id=RDIWOSET3Q2JKHRQ7HVUSVMNILHVEDQ7XYDL5DMWK37Q7YTRADQA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BdJtXzvjMoNoQUjtHETyADPcZBTzcFzH8YDd651t7eTiFb1wf): protocol: expected piece hash; context canceled; EOF
Jul 9 01:56:56.047756840 (376h46m30.818398175s) [noise=true , address=104.36.82.22:4532 , piece_id=3BRLUKXPLFBXIPKURV3BBSTA562UWWQBZ75HNYXEX75E74FKJIGQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cHUuyc42zpw5k9Lz5Qo9AW9DMvKFPxRNdKZDZtjFaeyzT4Nkp): protocol: expected piece hash; context canceled; EOF
Jul 9 01:56:56.047948587 (376h46m30.818589922s) [noise=true , address=104.36.82.21:4508 , piece_id=VVX5U234EZZNCZWIVYJVYBFDYG2SCKWS5HT5GM4H7CNNJQXDHT3Q, seg_pos={0 0}, upload=301417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1269NRnqpabU6H8q9vSCmE8yMzznD1L7dyZFmzCWhtzJaTjbXg3): protocol: write tcp 188.166.225.170:60172->104.36.82.21:4508: use of closed network connection; write tcp 188.166.225.170:60172->104.36.82.21:4508: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60172->104.36.82.21:4508: use of closed network connection
Jul 9 01:56:56.047987552 (376h46m30.818628887s) [noise=true , address=104.36.82.6:4880 , piece_id=NKXAOUHYMQ4WR6V6ZGQOKRZWLAGI44AF4SNUR4OKUNJNICD4UUNQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125NsQhq7kRmTjKS8nuc51PZddApFEi5C38FzC5pEX2QrWYKzSW): protocol: write tcp 188.166.225.170:38944->104.36.82.6:4880: use of closed network connection; write tcp 188.166.225.170:38944->104.36.82.6:4880: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38944->104.36.82.6:4880: use of closed network connection
Jul 9 01:56:56.047996138 (376h46m30.818637463s) [seg_pos={0 0}, upload=301417]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 01:56:56.048046284 (376h46m30.818687619s) [upload=301417]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 01:56:56.048050091 (376h46m30.818691416s) [upload=301417]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 01:56:56.339173192 (376h46m31.109814527s) [upload=301417]: Done uploading segments...
Jul 9 01:56:56.339190416 (376h46m31.109831751s) [upload=301417]: Upload finished. err: <nil>
Duration: 40.455804469s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [........X......................X.................X......X...] duration:
5.69865505s
FAIL (2025-07-09 02:04:43 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
nceled; EOF
Jul 9 02:00:49.060314467 (376h50m23.830955812s) [seg_pos={0 0}, upload=301481]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil>
Jul 9 02:00:49.060367628 (376h50m23.831008963s) [upload=301481]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:00:49.060371145 (376h50m23.83101248s) [upload=301481]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:00:49.357373945 (376h50m24.12801528s) [upload=301481]: Done uploading segments...
Jul 9 02:00:49.357395647 (376h50m24.128036982s) [upload=301481]: Upload finished. err: <nil>
Jul 9 02:02:01.314289870 (376h51m36.084931205s) [upload=301495]: Starting upload
Jul 9 02:02:01.314394990 (376h51m36.085036325s) [upload=301495]: Done starting upload
Jul 9 02:02:01.314484112 (376h51m36.085125457s) [upload=301495]: Uploading segments...
Jul 9 02:02:01.314699112 (376h51m36.085340457s) [upload=301495]: Got next segment. Inline: false
Jul 9 02:02:01.314747194 (376h51m36.085388529s) [upload=301495]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:02:01.314753927 (376h51m36.085395272s) [upload=301495]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:02:01.549200488 (376h51m36.319841873s) [seg_pos={0 0}, upload=301495]: Begin upload segment...
Jul 9 02:02:01.549266264 (376h51m36.319907609s) [seg_pos={0 0}, upload=301495]: Done begin upload segment.
Jul 9 02:02:01.549310128 (376h51m36.319951473s) [noise=true , address=104.36.82.7:4910 , piece_id=CPROZR2V6J7RKRZW6CNLA7P44DQVYH3VOZMYBBTEIBOL7MGINPDA, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.549336468 (376h51m36.319977823s) [noise=true , address=45.91.116.14:4190 , piece_id=ZAF4GOHABQQENBFRPE3HEMFTT474ULDEB5MZ5XDVVHFVTQID5AVA, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.549964509 (376h51m36.320605844s) [noise=true , address=104.36.82.13:4187 , piece_id=BLI6GHKTU3D5Q6FWTOS6UEPDLBDNCX2VEZCKAXIVAF32VWU7I6XQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.550253653 (376h51m36.320894988s) [noise=true , address=104.36.82.9:4006 , piece_id=XX7OUX5VBH5UVHELOP525HLHU5LA7O4B6DR6LKI7VYFWZYVAXHTQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.550518870 (376h51m36.321160225s) [noise=true , address=104.36.82.6:4874 , piece_id=UDKHL2V3C6ZYBR3FGHVIKQYJLPCOO3XLXYTSQJVLTGJ6I74IGGAA, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.550556472 (376h51m36.321197807s) [noise=true , address=45.91.116.10:4082 , piece_id=IRWOX4OQI5TB7FAT53RSX4C6DPNPUGZPPQXS6VFM2EE6FPZI4N4A, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.550837249 (376h51m36.321478584s) [noise=true , address=45.91.116.8:4048 , piece_id=UTXJQHFKWTANYVEEL2CICL5MQWVLUC4H4ZQ3IFXQD5O45RKB425A, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.551078060 (376h51m36.321719395s) [noise=true , address=45.91.116.7:4010 , piece_id=7CYJB5QPC3ZJZVGFNJKBZHOLSKPFBYMSWTJL37CQ73JIIM7LVA2A, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.551313831 (376h51m36.321955166s) [noise=true , address=77.90.31.11:4555 , piece_id=57OEP4BSHFGZQ5A7DJKJ4BNSO7ORPXPVMSN2I56YFREXE6N5RIPQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.551369758 (376h51m36.322011093s) [noise=true , address=45.91.116.11:4119 , piece_id=2TR3FC2ZYYNAPWL2GTL5EZVJW74NI5W275AYDNSZIME62TPILDBQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.551601652 (376h51m36.322242987s) [noise=true , address=104.36.82.5:4839 , piece_id=MM6CHXUZFEON5CG7GUKRLD7PGYKU3337UBU73DTHSKAKQUJUKYLQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.551803969 (376h51m36.322445304s) [noise=true , address=45.91.116.6:4973 , piece_id=2XDTRFYDWADB2U2DIT6VDWTHT6DVRPRNMCWJFW4Q7G7KXITLI6LQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.552051763 (376h51m36.322693098s) [noise=true , address=104.36.82.15:4258 , piece_id=KFL7XC6ILPLWJGYIGZVJ6X3ECS26XNNY6M4ZS3QHGXBPZG4XWILQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.552343200 (376h51m36.322984545s) [noise=true , address=45.91.116.13:4154 , piece_id=I2XLUYF36R6HKCEYHRUISRDMAPYP7CQHEXL7XB6IMG7GYO32W55Q, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.552474742 (376h51m36.323116077s) [noise=true , address=45.91.116.4:4939 , piece_id=QDXP222CYJXWF2NV5LKI7XOPMU7APMWDFQLWURFQG5BVJUNKNJBA, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.552718097 (376h51m36.323359433s) [noise=true , address=104.36.82.11:4114 , piece_id=R7MKGZKIKESIOB5CWCDO4KV2CNGN2EXTLGATSMK7Y2JMSFIB3DSA, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:03.251350348 (376h51m38.021991683s) [upload=301495]: Next returned nil segment
Jul 9 02:02:03.251367120 (376h51m38.022008455s) [upload=301495]: Waiting for error group managing segments...
Jul 9 02:02:03.536295419 (376h51m38.306936775s) [noise=true , address=104.36.82.9:4006 , piece_id=XX7OUX5VBH5UVHELOP525HLHU5LA7O4B6DR6LKI7VYFWZYVAXHTQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.585972469 (376h51m38.356613794s) [noise=true , address=45.91.116.8:4048 , piece_id=UTXJQHFKWTANYVEEL2CICL5MQWVLUC4H4ZQ3IFXQD5O45RKB425A, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.612355463 (376h51m38.382996798s) [noise=true , address=45.91.116.11:4119 , piece_id=2TR3FC2ZYYNAPWL2GTL5EZVJW74NI5W275AYDNSZIME62TPILDBQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.616140875 (376h51m38.38678221s) [noise=true , address=45.91.116.4:4939 , piece_id=QDXP222CYJXWF2NV5LKI7XOPMU7APMWDFQLWURFQG5BVJUNKNJBA, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.625063725 (376h51m38.395705061s) [noise=true , address=45.91.116.13:4154 , piece_id=I2XLUYF36R6HKCEYHRUISRDMAPYP7CQHEXL7XB6IMG7GYO32W55Q, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.627327436 (376h51m38.397968771s) [noise=true , address=77.90.31.11:4555 , piece_id=57OEP4BSHFGZQ5A7DJKJ4BNSO7ORPXPVMSN2I56YFREXE6N5RIPQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.680079459 (376h51m38.450720804s) [noise=true , address=45.91.116.14:4190 , piece_id=ZAF4GOHABQQENBFRPE3HEMFTT474ULDEB5MZ5XDVVHFVTQID5AVA, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.682275680 (376h51m38.452917015s) [noise=true , address=45.91.116.10:4082 , piece_id=IRWOX4OQI5TB7FAT53RSX4C6DPNPUGZPPQXS6VFM2EE6FPZI4N4A, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.693418348 (376h51m38.464059684s) [noise=true , address=45.91.116.6:4973 , piece_id=2XDTRFYDWADB2U2DIT6VDWTHT6DVRPRNMCWJFW4Q7G7KXITLI6LQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.714112589 (376h51m38.484753924s) [noise=true , address=45.91.116.7:4010 , piece_id=7CYJB5QPC3ZJZVGFNJKBZHOLSKPFBYMSWTJL37CQ73JIIM7LVA2A, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:24.743005877 (376h51m59.513647202s) [noise=true , address=104.36.82.6:4874 , piece_id=UDKHL2V3C6ZYBR3FGHVIKQYJLPCOO3XLXYTSQJVLTGJ6I74IGGAA, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:28.578275204 (376h52m3.348916539s) [noise=true , address=104.36.82.13:4187 , piece_id=BLI6GHKTU3D5Q6FWTOS6UEPDLBDNCX2VEZCKAXIVAF32VWU7I6XQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:30.664723094 (376h52m5.435364439s) [noise=true , address=104.36.82.11:4114 , piece_id=R7MKGZKIKESIOB5CWCDO4KV2CNGN2EXTLGATSMK7Y2JMSFIB3DSA, seg_pos={0 0}, upload=301495]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:02:30.665014852 (376h52m5.435656188s) [noise=true , address=104.36.82.5:4839 , piece_id=MM6CHXUZFEON5CG7GUKRLD7PGYKU3337UBU73DTHSKAKQUJUKYLQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 188.166.225.170:42438->104.36.82.5:4839: use of closed network connection; write tcp 188.166.225.170:42438->104.36.82.5:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42438->104.36.82.5:4839: use of closed network connection
Jul 9 02:02:30.665113180 (376h52m5.435754525s) [noise=true , address=104.36.82.7:4910 , piece_id=CPROZR2V6J7RKRZW6CNLA7P44DQVYH3VOZMYBBTEIBOL7MGINPDA, seg_pos={0 0}, upload=301495]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: write tcp 188.166.225.170:38368->104.36.82.7:4910: use of closed network connection; write tcp 188.166.225.170:38368->104.36.82.7:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38368->104.36.82.7:4910: use of closed network connection
Jul 9 02:02:30.665314215 (376h52m5.43595555s) [noise=true , address=104.36.82.15:4258 , piece_id=KFL7XC6ILPLWJGYIGZVJ6X3ECS26XNNY6M4ZS3QHGXBPZG4XWILQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 188.166.225.170:50202->104.36.82.15:4258: use of closed network connection; context deadline exceeded
Jul 9 02:02:30.665353560 (376h52m5.435994895s) [seg_pos={0 0}, upload=301495]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 02:02:30.900452118 (376h52m5.671093453s) [upload=301495]: Done uploading segments...
Jul 9 02:02:30.900486934 (376h52m5.67112827s) [upload=301495]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.698672022s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [.................X......X..........X.............X..........] duration:
43.920419329s
FAIL (2025-07-09 02:04:07 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
hRequestItem_SegmentBegin
Jul 9 01:56:00.629444016 (376h45m35.400085361s) [seg_pos={0 0}, upload=301417]: Begin upload segment...
Jul 9 01:56:00.629512196 (376h45m35.400153531s) [seg_pos={0 0}, upload=301417]: Done begin upload segment.
Jul 9 01:56:00.629572552 (376h45m35.400213887s) [noise=true , address=104.36.82.22:4532 , piece_id=3BRLUKXPLFBXIPKURV3BBSTA562UWWQBZ75HNYXEX75E74FKJIGQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.629579284 (376h45m35.400220619s) [noise=true , address=77.90.31.10:4539 , piece_id=OI7V6HU5WMYZSQCS5PAN6OUCAJY4UGUZVTESMLB4LHOQJVMKJLHQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.629690497 (376h45m35.400331832s) [noise=true , address=104.36.82.17:4333 , piece_id=TPALHRSYWC3XKW77CPA36EZOOJHPLUEN4V6DWFN5NHLIAFSN5KRQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630082818 (376h45m35.400724153s) [noise=true , address=104.36.82.6:4880 , piece_id=NKXAOUHYMQ4WR6V6ZGQOKRZWLAGI44AF4SNUR4OKUNJNICD4UUNQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630158242 (376h45m35.400799577s) [noise=true , address=79.127.201.214:4308 , piece_id=V4LWWG3RM4JTLZGNZ3XJDHEOZO7XW5R7BHQQJ6Z4Y47PZCEZHN6A, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630428680 (376h45m35.401070025s) [noise=true , address=77.90.31.12:4615 , piece_id=GLIW3756SYGXSX44VFMMDNYVEJPS6O4ZBNS53OJRV67QNGMGLPUA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630538129 (376h45m35.401179464s) [noise=true , address=104.36.82.60:4364 , piece_id=7WZJU4UIO4FLPUNWUH2PVIIH3ZWWVZCP2D2UUSCNBCIS727VVWPA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630862820 (376h45m35.401504165s) [noise=true , address=104.36.82.38:4708 , piece_id=RDIWOSET3Q2JKHRQ7HVUSVMNILHVEDQ7XYDL5DMWK37Q7YTRADQA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.631210084 (376h45m35.401851419s) [noise=true , address=104.36.82.21:4508 , piece_id=VVX5U234EZZNCZWIVYJVYBFDYG2SCKWS5HT5GM4H7CNNJQXDHT3Q, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.631555495 (376h45m35.40219684s) [noise=true , address=77.90.31.16:4743 , piece_id=2VSOXKLN6Q434ET7JYLRSU6BW7PIKNZ43MZOQ4HQL6MDP3WUJOZA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.631989165 (376h45m35.4026305s) [noise=true , address=104.36.82.23:4473 , piece_id=V3RAX7LZBU744EQQXDEG6Z25UUGQUFSSQCWCXNZKNGCTHD375GCA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.632258450 (376h45m35.402899785s) [noise=true , address=77.90.31.19:4850 , piece_id=DNYDFQHORBVHQTPKFLWEGQF6QRLE6IIE6YGTRP6SS63CPWVKRFFA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.632521984 (376h45m35.403163309s) [noise=true , address=45.91.116.5:4322 , piece_id=QMOE7Z74V4HHQO7LIN6KAJHEUSDKIAVC4XLFH3AOJMC2ILP6I4FA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.632835304 (376h45m35.403476649s) [noise=true , address=104.36.82.3:4791 , piece_id=NG7KRWBROBZWNP63A4734KWSRNGFVGLLXFZ5PGXZDJXFDTZNCZLQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.633120529 (376h45m35.403761864s) [noise=true , address=77.90.31.14:4692 , piece_id=NHO2352W7MMVGRS3MNPWSEFHWBOK73SQ6JQJEWZ5TV5UDFY44H5Q, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.636119667 (376h45m35.406761002s) [noise=true , address=104.36.82.2:4757 , piece_id=JMJIECO7WCFHWASHRCTEAG6VCU4S25SEETORWGJJQ63VAW533YJA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.636833744 (376h45m35.407475079s) [noise=true , address=77.90.31.15:4712 , piece_id=3DN3NPUYRN4AIBYBV2UCGE4TTYXD2WJRLHDMQEFIU36HBTYDOJ5Q, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.637112587 (376h45m35.407753922s) [noise=true , address=77.90.31.13:4643 , piece_id=PO24YVCOVVNORI7NRD3CDUC4ZPRULAXUESGBEDHJZYYDJ2ZUPQDA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:02.137197163 (376h45m36.907838498s) [upload=301417]: Next returned nil segment
Jul 9 01:56:02.137214166 (376h45m36.907855501s) [upload=301417]: Waiting for error group managing segments...
Jul 9 01:56:02.521317983 (376h45m37.291959318s) [noise=true , address=79.127.201.214:4308 , piece_id=V4LWWG3RM4JTLZGNZ3XJDHEOZO7XW5R7BHQQJ6Z4Y47PZCEZHN6A, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.567664820 (376h45m37.338306156s) [noise=true , address=77.90.31.14:4692 , piece_id=NHO2352W7MMVGRS3MNPWSEFHWBOK73SQ6JQJEWZ5TV5UDFY44H5Q, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.568760968 (376h45m37.339402303s) [noise=true , address=77.90.31.10:4539 , piece_id=OI7V6HU5WMYZSQCS5PAN6OUCAJY4UGUZVTESMLB4LHOQJVMKJLHQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.573315612 (376h45m37.343956947s) [noise=true , address=77.90.31.15:4712 , piece_id=3DN3NPUYRN4AIBYBV2UCGE4TTYXD2WJRLHDMQEFIU36HBTYDOJ5Q, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.577761728 (376h45m37.348403053s) [noise=true , address=77.90.31.19:4850 , piece_id=DNYDFQHORBVHQTPKFLWEGQF6QRLE6IIE6YGTRP6SS63CPWVKRFFA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.579676210 (376h45m37.350317545s) [noise=true , address=77.90.31.12:4615 , piece_id=GLIW3756SYGXSX44VFMMDNYVEJPS6O4ZBNS53OJRV67QNGMGLPUA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.595175322 (376h45m37.365816657s) [noise=true , address=77.90.31.16:4743 , piece_id=2VSOXKLN6Q434ET7JYLRSU6BW7PIKNZ43MZOQ4HQL6MDP3WUJOZA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.601273909 (376h45m37.371915244s) [noise=true , address=77.90.31.13:4643 , piece_id=PO24YVCOVVNORI7NRD3CDUC4ZPRULAXUESGBEDHJZYYDJ2ZUPQDA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:07.782538378 (376h45m42.553179714s) [noise=true , address=45.91.116.5:4322 , piece_id=QMOE7Z74V4HHQO7LIN6KAJHEUSDKIAVC4XLFH3AOJMC2ILP6I4FA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:22.662088184 (376h45m57.432729519s) [noise=true , address=104.36.82.17:4333 , piece_id=TPALHRSYWC3XKW77CPA36EZOOJHPLUEN4V6DWFN5NHLIAFSN5KRQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:24.053592013 (376h45m58.824233348s) [noise=true , address=104.36.82.2:4757 , piece_id=JMJIECO7WCFHWASHRCTEAG6VCU4S25SEETORWGJJQ63VAW533YJA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:44.900784468 (376h46m19.671425793s) [noise=true , address=104.36.82.23:4473 , piece_id=V3RAX7LZBU744EQQXDEG6Z25UUGQUFSSQCWCXNZKNGCTHD375GCA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:47.726372911 (376h46m22.497014236s) [noise=true , address=104.36.82.60:4364 , piece_id=7WZJU4UIO4FLPUNWUH2PVIIH3ZWWVZCP2D2UUSCNBCIS727VVWPA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:56.047624046 (376h46m30.818265381s) [noise=true , address=104.36.82.3:4791 , piece_id=NG7KRWBROBZWNP63A4734KWSRNGFVGLLXFZ5PGXZDJXFDTZNCZLQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:56.047644345 (376h46m30.81828568s) [seg_pos={0 0}, upload=301417]: Segment reached optimal threshold of 14 pieces.
Jul 9 01:56:56.047718136 (376h46m30.818359471s) [noise=true , address=104.36.82.38:4708 , piece_id=RDIWOSET3Q2JKHRQ7HVUSVMNILHVEDQ7XYDL5DMWK37Q7YTRADQA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BdJtXzvjMoNoQUjtHETyADPcZBTzcFzH8YDd651t7eTiFb1wf): protocol: expected piece hash; context canceled; EOF
Jul 9 01:56:56.047756840 (376h46m30.818398175s) [noise=true , address=104.36.82.22:4532 , piece_id=3BRLUKXPLFBXIPKURV3BBSTA562UWWQBZ75HNYXEX75E74FKJIGQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cHUuyc42zpw5k9Lz5Qo9AW9DMvKFPxRNdKZDZtjFaeyzT4Nkp): protocol: expected piece hash; context canceled; EOF
Jul 9 01:56:56.047948587 (376h46m30.818589922s) [noise=true , address=104.36.82.21:4508 , piece_id=VVX5U234EZZNCZWIVYJVYBFDYG2SCKWS5HT5GM4H7CNNJQXDHT3Q, seg_pos={0 0}, upload=301417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1269NRnqpabU6H8q9vSCmE8yMzznD1L7dyZFmzCWhtzJaTjbXg3): protocol: write tcp 188.166.225.170:60172->104.36.82.21:4508: use of closed network connection; write tcp 188.166.225.170:60172->104.36.82.21:4508: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60172->104.36.82.21:4508: use of closed network connection
Jul 9 01:56:56.047987552 (376h46m30.818628887s) [noise=true , address=104.36.82.6:4880 , piece_id=NKXAOUHYMQ4WR6V6ZGQOKRZWLAGI44AF4SNUR4OKUNJNICD4UUNQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125NsQhq7kRmTjKS8nuc51PZddApFEi5C38FzC5pEX2QrWYKzSW): protocol: write tcp 188.166.225.170:38944->104.36.82.6:4880: use of closed network connection; write tcp 188.166.225.170:38944->104.36.82.6:4880: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38944->104.36.82.6:4880: use of closed network connection
Jul 9 01:56:56.047996138 (376h46m30.818637463s) [seg_pos={0 0}, upload=301417]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 01:56:56.048046284 (376h46m30.818687619s) [upload=301417]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 01:56:56.048050091 (376h46m30.818691416s) [upload=301417]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 01:56:56.339173192 (376h46m31.109814527s) [upload=301417]: Done uploading segments...
Jul 9 01:56:56.339190416 (376h46m31.109831751s) [upload=301417]: Upload finished. err: <nil>
Duration: 43.920440651s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
17 out of 60 is failed (threshold=3) [......X......XX.XX..X......XXX....XX.....XX...X.XX........X.] duration:
31.735888812s
FAIL (2025-07-09 02:04:03 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 31.735899873s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [..X.X..X...X..X............X......X..X...X.X.X.X...X........] duration:
5.965169936s
FAIL (2025-07-09 02:03:50 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.965198281s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.........X......................X.................X......X..] duration:
5.718531053s
FAIL (2025-07-09 02:03:31 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
nceled; EOF
Jul 9 02:00:49.060314467 (376h50m23.830955812s) [seg_pos={0 0}, upload=301481]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil>
Jul 9 02:00:49.060367628 (376h50m23.831008963s) [upload=301481]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:00:49.060371145 (376h50m23.83101248s) [upload=301481]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:00:49.357373945 (376h50m24.12801528s) [upload=301481]: Done uploading segments...
Jul 9 02:00:49.357395647 (376h50m24.128036982s) [upload=301481]: Upload finished. err: <nil>
Jul 9 02:02:01.314289870 (376h51m36.084931205s) [upload=301495]: Starting upload
Jul 9 02:02:01.314394990 (376h51m36.085036325s) [upload=301495]: Done starting upload
Jul 9 02:02:01.314484112 (376h51m36.085125457s) [upload=301495]: Uploading segments...
Jul 9 02:02:01.314699112 (376h51m36.085340457s) [upload=301495]: Got next segment. Inline: false
Jul 9 02:02:01.314747194 (376h51m36.085388529s) [upload=301495]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:02:01.314753927 (376h51m36.085395272s) [upload=301495]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:02:01.549200488 (376h51m36.319841873s) [seg_pos={0 0}, upload=301495]: Begin upload segment...
Jul 9 02:02:01.549266264 (376h51m36.319907609s) [seg_pos={0 0}, upload=301495]: Done begin upload segment.
Jul 9 02:02:01.549310128 (376h51m36.319951473s) [noise=true , address=104.36.82.7:4910 , piece_id=CPROZR2V6J7RKRZW6CNLA7P44DQVYH3VOZMYBBTEIBOL7MGINPDA, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.549336468 (376h51m36.319977823s) [noise=true , address=45.91.116.14:4190 , piece_id=ZAF4GOHABQQENBFRPE3HEMFTT474ULDEB5MZ5XDVVHFVTQID5AVA, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.549964509 (376h51m36.320605844s) [noise=true , address=104.36.82.13:4187 , piece_id=BLI6GHKTU3D5Q6FWTOS6UEPDLBDNCX2VEZCKAXIVAF32VWU7I6XQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.550253653 (376h51m36.320894988s) [noise=true , address=104.36.82.9:4006 , piece_id=XX7OUX5VBH5UVHELOP525HLHU5LA7O4B6DR6LKI7VYFWZYVAXHTQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.550518870 (376h51m36.321160225s) [noise=true , address=104.36.82.6:4874 , piece_id=UDKHL2V3C6ZYBR3FGHVIKQYJLPCOO3XLXYTSQJVLTGJ6I74IGGAA, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.550556472 (376h51m36.321197807s) [noise=true , address=45.91.116.10:4082 , piece_id=IRWOX4OQI5TB7FAT53RSX4C6DPNPUGZPPQXS6VFM2EE6FPZI4N4A, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.550837249 (376h51m36.321478584s) [noise=true , address=45.91.116.8:4048 , piece_id=UTXJQHFKWTANYVEEL2CICL5MQWVLUC4H4ZQ3IFXQD5O45RKB425A, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.551078060 (376h51m36.321719395s) [noise=true , address=45.91.116.7:4010 , piece_id=7CYJB5QPC3ZJZVGFNJKBZHOLSKPFBYMSWTJL37CQ73JIIM7LVA2A, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.551313831 (376h51m36.321955166s) [noise=true , address=77.90.31.11:4555 , piece_id=57OEP4BSHFGZQ5A7DJKJ4BNSO7ORPXPVMSN2I56YFREXE6N5RIPQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.551369758 (376h51m36.322011093s) [noise=true , address=45.91.116.11:4119 , piece_id=2TR3FC2ZYYNAPWL2GTL5EZVJW74NI5W275AYDNSZIME62TPILDBQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.551601652 (376h51m36.322242987s) [noise=true , address=104.36.82.5:4839 , piece_id=MM6CHXUZFEON5CG7GUKRLD7PGYKU3337UBU73DTHSKAKQUJUKYLQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.551803969 (376h51m36.322445304s) [noise=true , address=45.91.116.6:4973 , piece_id=2XDTRFYDWADB2U2DIT6VDWTHT6DVRPRNMCWJFW4Q7G7KXITLI6LQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.552051763 (376h51m36.322693098s) [noise=true , address=104.36.82.15:4258 , piece_id=KFL7XC6ILPLWJGYIGZVJ6X3ECS26XNNY6M4ZS3QHGXBPZG4XWILQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.552343200 (376h51m36.322984545s) [noise=true , address=45.91.116.13:4154 , piece_id=I2XLUYF36R6HKCEYHRUISRDMAPYP7CQHEXL7XB6IMG7GYO32W55Q, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.552474742 (376h51m36.323116077s) [noise=true , address=45.91.116.4:4939 , piece_id=QDXP222CYJXWF2NV5LKI7XOPMU7APMWDFQLWURFQG5BVJUNKNJBA, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.552718097 (376h51m36.323359433s) [noise=true , address=104.36.82.11:4114 , piece_id=R7MKGZKIKESIOB5CWCDO4KV2CNGN2EXTLGATSMK7Y2JMSFIB3DSA, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:03.251350348 (376h51m38.021991683s) [upload=301495]: Next returned nil segment
Jul 9 02:02:03.251367120 (376h51m38.022008455s) [upload=301495]: Waiting for error group managing segments...
Jul 9 02:02:03.536295419 (376h51m38.306936775s) [noise=true , address=104.36.82.9:4006 , piece_id=XX7OUX5VBH5UVHELOP525HLHU5LA7O4B6DR6LKI7VYFWZYVAXHTQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.585972469 (376h51m38.356613794s) [noise=true , address=45.91.116.8:4048 , piece_id=UTXJQHFKWTANYVEEL2CICL5MQWVLUC4H4ZQ3IFXQD5O45RKB425A, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.612355463 (376h51m38.382996798s) [noise=true , address=45.91.116.11:4119 , piece_id=2TR3FC2ZYYNAPWL2GTL5EZVJW74NI5W275AYDNSZIME62TPILDBQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.616140875 (376h51m38.38678221s) [noise=true , address=45.91.116.4:4939 , piece_id=QDXP222CYJXWF2NV5LKI7XOPMU7APMWDFQLWURFQG5BVJUNKNJBA, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.625063725 (376h51m38.395705061s) [noise=true , address=45.91.116.13:4154 , piece_id=I2XLUYF36R6HKCEYHRUISRDMAPYP7CQHEXL7XB6IMG7GYO32W55Q, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.627327436 (376h51m38.397968771s) [noise=true , address=77.90.31.11:4555 , piece_id=57OEP4BSHFGZQ5A7DJKJ4BNSO7ORPXPVMSN2I56YFREXE6N5RIPQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.680079459 (376h51m38.450720804s) [noise=true , address=45.91.116.14:4190 , piece_id=ZAF4GOHABQQENBFRPE3HEMFTT474ULDEB5MZ5XDVVHFVTQID5AVA, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.682275680 (376h51m38.452917015s) [noise=true , address=45.91.116.10:4082 , piece_id=IRWOX4OQI5TB7FAT53RSX4C6DPNPUGZPPQXS6VFM2EE6FPZI4N4A, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.693418348 (376h51m38.464059684s) [noise=true , address=45.91.116.6:4973 , piece_id=2XDTRFYDWADB2U2DIT6VDWTHT6DVRPRNMCWJFW4Q7G7KXITLI6LQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.714112589 (376h51m38.484753924s) [noise=true , address=45.91.116.7:4010 , piece_id=7CYJB5QPC3ZJZVGFNJKBZHOLSKPFBYMSWTJL37CQ73JIIM7LVA2A, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:24.743005877 (376h51m59.513647202s) [noise=true , address=104.36.82.6:4874 , piece_id=UDKHL2V3C6ZYBR3FGHVIKQYJLPCOO3XLXYTSQJVLTGJ6I74IGGAA, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:28.578275204 (376h52m3.348916539s) [noise=true , address=104.36.82.13:4187 , piece_id=BLI6GHKTU3D5Q6FWTOS6UEPDLBDNCX2VEZCKAXIVAF32VWU7I6XQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:30.664723094 (376h52m5.435364439s) [noise=true , address=104.36.82.11:4114 , piece_id=R7MKGZKIKESIOB5CWCDO4KV2CNGN2EXTLGATSMK7Y2JMSFIB3DSA, seg_pos={0 0}, upload=301495]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:02:30.665014852 (376h52m5.435656188s) [noise=true , address=104.36.82.5:4839 , piece_id=MM6CHXUZFEON5CG7GUKRLD7PGYKU3337UBU73DTHSKAKQUJUKYLQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 188.166.225.170:42438->104.36.82.5:4839: use of closed network connection; write tcp 188.166.225.170:42438->104.36.82.5:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42438->104.36.82.5:4839: use of closed network connection
Jul 9 02:02:30.665113180 (376h52m5.435754525s) [noise=true , address=104.36.82.7:4910 , piece_id=CPROZR2V6J7RKRZW6CNLA7P44DQVYH3VOZMYBBTEIBOL7MGINPDA, seg_pos={0 0}, upload=301495]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: write tcp 188.166.225.170:38368->104.36.82.7:4910: use of closed network connection; write tcp 188.166.225.170:38368->104.36.82.7:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38368->104.36.82.7:4910: use of closed network connection
Jul 9 02:02:30.665314215 (376h52m5.43595555s) [noise=true , address=104.36.82.15:4258 , piece_id=KFL7XC6ILPLWJGYIGZVJ6X3ECS26XNNY6M4ZS3QHGXBPZG4XWILQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 188.166.225.170:50202->104.36.82.15:4258: use of closed network connection; context deadline exceeded
Jul 9 02:02:30.665353560 (376h52m5.435994895s) [seg_pos={0 0}, upload=301495]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 02:02:30.900452118 (376h52m5.671093453s) [upload=301495]: Done uploading segments...
Jul 9 02:02:30.900486934 (376h52m5.67112827s) [upload=301495]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.718543207s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
17 out of 60 is failed (threshold=3) [.......X......XX.XX..X......XXX....XX.....XX...X.XX........X] duration:
1m0.000266945s
FAIL (2025-07-09 02:03:31 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.000280352s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [..........X......................X.................X......X.] duration:
29.444705613s
FAIL (2025-07-09 02:03:25 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
nceled; EOF
Jul 9 02:00:49.060314467 (376h50m23.830955812s) [seg_pos={0 0}, upload=301481]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil>
Jul 9 02:00:49.060367628 (376h50m23.831008963s) [upload=301481]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:00:49.060371145 (376h50m23.83101248s) [upload=301481]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:00:49.357373945 (376h50m24.12801528s) [upload=301481]: Done uploading segments...
Jul 9 02:00:49.357395647 (376h50m24.128036982s) [upload=301481]: Upload finished. err: <nil>
Jul 9 02:02:01.314289870 (376h51m36.084931205s) [upload=301495]: Starting upload
Jul 9 02:02:01.314394990 (376h51m36.085036325s) [upload=301495]: Done starting upload
Jul 9 02:02:01.314484112 (376h51m36.085125457s) [upload=301495]: Uploading segments...
Jul 9 02:02:01.314699112 (376h51m36.085340457s) [upload=301495]: Got next segment. Inline: false
Jul 9 02:02:01.314747194 (376h51m36.085388529s) [upload=301495]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:02:01.314753927 (376h51m36.085395272s) [upload=301495]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:02:01.549200488 (376h51m36.319841873s) [seg_pos={0 0}, upload=301495]: Begin upload segment...
Jul 9 02:02:01.549266264 (376h51m36.319907609s) [seg_pos={0 0}, upload=301495]: Done begin upload segment.
Jul 9 02:02:01.549310128 (376h51m36.319951473s) [noise=true , address=104.36.82.7:4910 , piece_id=CPROZR2V6J7RKRZW6CNLA7P44DQVYH3VOZMYBBTEIBOL7MGINPDA, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.549336468 (376h51m36.319977823s) [noise=true , address=45.91.116.14:4190 , piece_id=ZAF4GOHABQQENBFRPE3HEMFTT474ULDEB5MZ5XDVVHFVTQID5AVA, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.549964509 (376h51m36.320605844s) [noise=true , address=104.36.82.13:4187 , piece_id=BLI6GHKTU3D5Q6FWTOS6UEPDLBDNCX2VEZCKAXIVAF32VWU7I6XQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.550253653 (376h51m36.320894988s) [noise=true , address=104.36.82.9:4006 , piece_id=XX7OUX5VBH5UVHELOP525HLHU5LA7O4B6DR6LKI7VYFWZYVAXHTQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.550518870 (376h51m36.321160225s) [noise=true , address=104.36.82.6:4874 , piece_id=UDKHL2V3C6ZYBR3FGHVIKQYJLPCOO3XLXYTSQJVLTGJ6I74IGGAA, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.550556472 (376h51m36.321197807s) [noise=true , address=45.91.116.10:4082 , piece_id=IRWOX4OQI5TB7FAT53RSX4C6DPNPUGZPPQXS6VFM2EE6FPZI4N4A, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.550837249 (376h51m36.321478584s) [noise=true , address=45.91.116.8:4048 , piece_id=UTXJQHFKWTANYVEEL2CICL5MQWVLUC4H4ZQ3IFXQD5O45RKB425A, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.551078060 (376h51m36.321719395s) [noise=true , address=45.91.116.7:4010 , piece_id=7CYJB5QPC3ZJZVGFNJKBZHOLSKPFBYMSWTJL37CQ73JIIM7LVA2A, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.551313831 (376h51m36.321955166s) [noise=true , address=77.90.31.11:4555 , piece_id=57OEP4BSHFGZQ5A7DJKJ4BNSO7ORPXPVMSN2I56YFREXE6N5RIPQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.551369758 (376h51m36.322011093s) [noise=true , address=45.91.116.11:4119 , piece_id=2TR3FC2ZYYNAPWL2GTL5EZVJW74NI5W275AYDNSZIME62TPILDBQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.551601652 (376h51m36.322242987s) [noise=true , address=104.36.82.5:4839 , piece_id=MM6CHXUZFEON5CG7GUKRLD7PGYKU3337UBU73DTHSKAKQUJUKYLQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.551803969 (376h51m36.322445304s) [noise=true , address=45.91.116.6:4973 , piece_id=2XDTRFYDWADB2U2DIT6VDWTHT6DVRPRNMCWJFW4Q7G7KXITLI6LQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.552051763 (376h51m36.322693098s) [noise=true , address=104.36.82.15:4258 , piece_id=KFL7XC6ILPLWJGYIGZVJ6X3ECS26XNNY6M4ZS3QHGXBPZG4XWILQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.552343200 (376h51m36.322984545s) [noise=true , address=45.91.116.13:4154 , piece_id=I2XLUYF36R6HKCEYHRUISRDMAPYP7CQHEXL7XB6IMG7GYO32W55Q, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.552474742 (376h51m36.323116077s) [noise=true , address=45.91.116.4:4939 , piece_id=QDXP222CYJXWF2NV5LKI7XOPMU7APMWDFQLWURFQG5BVJUNKNJBA, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.552718097 (376h51m36.323359433s) [noise=true , address=104.36.82.11:4114 , piece_id=R7MKGZKIKESIOB5CWCDO4KV2CNGN2EXTLGATSMK7Y2JMSFIB3DSA, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:03.251350348 (376h51m38.021991683s) [upload=301495]: Next returned nil segment
Jul 9 02:02:03.251367120 (376h51m38.022008455s) [upload=301495]: Waiting for error group managing segments...
Jul 9 02:02:03.536295419 (376h51m38.306936775s) [noise=true , address=104.36.82.9:4006 , piece_id=XX7OUX5VBH5UVHELOP525HLHU5LA7O4B6DR6LKI7VYFWZYVAXHTQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.585972469 (376h51m38.356613794s) [noise=true , address=45.91.116.8:4048 , piece_id=UTXJQHFKWTANYVEEL2CICL5MQWVLUC4H4ZQ3IFXQD5O45RKB425A, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.612355463 (376h51m38.382996798s) [noise=true , address=45.91.116.11:4119 , piece_id=2TR3FC2ZYYNAPWL2GTL5EZVJW74NI5W275AYDNSZIME62TPILDBQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.616140875 (376h51m38.38678221s) [noise=true , address=45.91.116.4:4939 , piece_id=QDXP222CYJXWF2NV5LKI7XOPMU7APMWDFQLWURFQG5BVJUNKNJBA, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.625063725 (376h51m38.395705061s) [noise=true , address=45.91.116.13:4154 , piece_id=I2XLUYF36R6HKCEYHRUISRDMAPYP7CQHEXL7XB6IMG7GYO32W55Q, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.627327436 (376h51m38.397968771s) [noise=true , address=77.90.31.11:4555 , piece_id=57OEP4BSHFGZQ5A7DJKJ4BNSO7ORPXPVMSN2I56YFREXE6N5RIPQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.680079459 (376h51m38.450720804s) [noise=true , address=45.91.116.14:4190 , piece_id=ZAF4GOHABQQENBFRPE3HEMFTT474ULDEB5MZ5XDVVHFVTQID5AVA, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.682275680 (376h51m38.452917015s) [noise=true , address=45.91.116.10:4082 , piece_id=IRWOX4OQI5TB7FAT53RSX4C6DPNPUGZPPQXS6VFM2EE6FPZI4N4A, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.693418348 (376h51m38.464059684s) [noise=true , address=45.91.116.6:4973 , piece_id=2XDTRFYDWADB2U2DIT6VDWTHT6DVRPRNMCWJFW4Q7G7KXITLI6LQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.714112589 (376h51m38.484753924s) [noise=true , address=45.91.116.7:4010 , piece_id=7CYJB5QPC3ZJZVGFNJKBZHOLSKPFBYMSWTJL37CQ73JIIM7LVA2A, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:24.743005877 (376h51m59.513647202s) [noise=true , address=104.36.82.6:4874 , piece_id=UDKHL2V3C6ZYBR3FGHVIKQYJLPCOO3XLXYTSQJVLTGJ6I74IGGAA, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:28.578275204 (376h52m3.348916539s) [noise=true , address=104.36.82.13:4187 , piece_id=BLI6GHKTU3D5Q6FWTOS6UEPDLBDNCX2VEZCKAXIVAF32VWU7I6XQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:30.664723094 (376h52m5.435364439s) [noise=true , address=104.36.82.11:4114 , piece_id=R7MKGZKIKESIOB5CWCDO4KV2CNGN2EXTLGATSMK7Y2JMSFIB3DSA, seg_pos={0 0}, upload=301495]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:02:30.665014852 (376h52m5.435656188s) [noise=true , address=104.36.82.5:4839 , piece_id=MM6CHXUZFEON5CG7GUKRLD7PGYKU3337UBU73DTHSKAKQUJUKYLQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 188.166.225.170:42438->104.36.82.5:4839: use of closed network connection; write tcp 188.166.225.170:42438->104.36.82.5:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42438->104.36.82.5:4839: use of closed network connection
Jul 9 02:02:30.665113180 (376h52m5.435754525s) [noise=true , address=104.36.82.7:4910 , piece_id=CPROZR2V6J7RKRZW6CNLA7P44DQVYH3VOZMYBBTEIBOL7MGINPDA, seg_pos={0 0}, upload=301495]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: write tcp 188.166.225.170:38368->104.36.82.7:4910: use of closed network connection; write tcp 188.166.225.170:38368->104.36.82.7:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38368->104.36.82.7:4910: use of closed network connection
Jul 9 02:02:30.665314215 (376h52m5.43595555s) [noise=true , address=104.36.82.15:4258 , piece_id=KFL7XC6ILPLWJGYIGZVJ6X3ECS26XNNY6M4ZS3QHGXBPZG4XWILQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 188.166.225.170:50202->104.36.82.15:4258: use of closed network connection; context deadline exceeded
Jul 9 02:02:30.665353560 (376h52m5.435994895s) [seg_pos={0 0}, upload=301495]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 02:02:30.900452118 (376h52m5.671093453s) [upload=301495]: Done uploading segments...
Jul 9 02:02:30.900486934 (376h52m5.67112827s) [upload=301495]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 29.444732003s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [..................X......X..........X.............X.........] duration:
42.669215331s
FAIL (2025-07-09 02:03:22 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
hRequestItem_SegmentBegin
Jul 9 01:56:00.629444016 (376h45m35.400085361s) [seg_pos={0 0}, upload=301417]: Begin upload segment...
Jul 9 01:56:00.629512196 (376h45m35.400153531s) [seg_pos={0 0}, upload=301417]: Done begin upload segment.
Jul 9 01:56:00.629572552 (376h45m35.400213887s) [noise=true , address=104.36.82.22:4532 , piece_id=3BRLUKXPLFBXIPKURV3BBSTA562UWWQBZ75HNYXEX75E74FKJIGQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.629579284 (376h45m35.400220619s) [noise=true , address=77.90.31.10:4539 , piece_id=OI7V6HU5WMYZSQCS5PAN6OUCAJY4UGUZVTESMLB4LHOQJVMKJLHQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.629690497 (376h45m35.400331832s) [noise=true , address=104.36.82.17:4333 , piece_id=TPALHRSYWC3XKW77CPA36EZOOJHPLUEN4V6DWFN5NHLIAFSN5KRQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630082818 (376h45m35.400724153s) [noise=true , address=104.36.82.6:4880 , piece_id=NKXAOUHYMQ4WR6V6ZGQOKRZWLAGI44AF4SNUR4OKUNJNICD4UUNQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630158242 (376h45m35.400799577s) [noise=true , address=79.127.201.214:4308 , piece_id=V4LWWG3RM4JTLZGNZ3XJDHEOZO7XW5R7BHQQJ6Z4Y47PZCEZHN6A, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630428680 (376h45m35.401070025s) [noise=true , address=77.90.31.12:4615 , piece_id=GLIW3756SYGXSX44VFMMDNYVEJPS6O4ZBNS53OJRV67QNGMGLPUA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630538129 (376h45m35.401179464s) [noise=true , address=104.36.82.60:4364 , piece_id=7WZJU4UIO4FLPUNWUH2PVIIH3ZWWVZCP2D2UUSCNBCIS727VVWPA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630862820 (376h45m35.401504165s) [noise=true , address=104.36.82.38:4708 , piece_id=RDIWOSET3Q2JKHRQ7HVUSVMNILHVEDQ7XYDL5DMWK37Q7YTRADQA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.631210084 (376h45m35.401851419s) [noise=true , address=104.36.82.21:4508 , piece_id=VVX5U234EZZNCZWIVYJVYBFDYG2SCKWS5HT5GM4H7CNNJQXDHT3Q, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.631555495 (376h45m35.40219684s) [noise=true , address=77.90.31.16:4743 , piece_id=2VSOXKLN6Q434ET7JYLRSU6BW7PIKNZ43MZOQ4HQL6MDP3WUJOZA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.631989165 (376h45m35.4026305s) [noise=true , address=104.36.82.23:4473 , piece_id=V3RAX7LZBU744EQQXDEG6Z25UUGQUFSSQCWCXNZKNGCTHD375GCA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.632258450 (376h45m35.402899785s) [noise=true , address=77.90.31.19:4850 , piece_id=DNYDFQHORBVHQTPKFLWEGQF6QRLE6IIE6YGTRP6SS63CPWVKRFFA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.632521984 (376h45m35.403163309s) [noise=true , address=45.91.116.5:4322 , piece_id=QMOE7Z74V4HHQO7LIN6KAJHEUSDKIAVC4XLFH3AOJMC2ILP6I4FA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.632835304 (376h45m35.403476649s) [noise=true , address=104.36.82.3:4791 , piece_id=NG7KRWBROBZWNP63A4734KWSRNGFVGLLXFZ5PGXZDJXFDTZNCZLQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.633120529 (376h45m35.403761864s) [noise=true , address=77.90.31.14:4692 , piece_id=NHO2352W7MMVGRS3MNPWSEFHWBOK73SQ6JQJEWZ5TV5UDFY44H5Q, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.636119667 (376h45m35.406761002s) [noise=true , address=104.36.82.2:4757 , piece_id=JMJIECO7WCFHWASHRCTEAG6VCU4S25SEETORWGJJQ63VAW533YJA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.636833744 (376h45m35.407475079s) [noise=true , address=77.90.31.15:4712 , piece_id=3DN3NPUYRN4AIBYBV2UCGE4TTYXD2WJRLHDMQEFIU36HBTYDOJ5Q, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.637112587 (376h45m35.407753922s) [noise=true , address=77.90.31.13:4643 , piece_id=PO24YVCOVVNORI7NRD3CDUC4ZPRULAXUESGBEDHJZYYDJ2ZUPQDA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:02.137197163 (376h45m36.907838498s) [upload=301417]: Next returned nil segment
Jul 9 01:56:02.137214166 (376h45m36.907855501s) [upload=301417]: Waiting for error group managing segments...
Jul 9 01:56:02.521317983 (376h45m37.291959318s) [noise=true , address=79.127.201.214:4308 , piece_id=V4LWWG3RM4JTLZGNZ3XJDHEOZO7XW5R7BHQQJ6Z4Y47PZCEZHN6A, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.567664820 (376h45m37.338306156s) [noise=true , address=77.90.31.14:4692 , piece_id=NHO2352W7MMVGRS3MNPWSEFHWBOK73SQ6JQJEWZ5TV5UDFY44H5Q, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.568760968 (376h45m37.339402303s) [noise=true , address=77.90.31.10:4539 , piece_id=OI7V6HU5WMYZSQCS5PAN6OUCAJY4UGUZVTESMLB4LHOQJVMKJLHQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.573315612 (376h45m37.343956947s) [noise=true , address=77.90.31.15:4712 , piece_id=3DN3NPUYRN4AIBYBV2UCGE4TTYXD2WJRLHDMQEFIU36HBTYDOJ5Q, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.577761728 (376h45m37.348403053s) [noise=true , address=77.90.31.19:4850 , piece_id=DNYDFQHORBVHQTPKFLWEGQF6QRLE6IIE6YGTRP6SS63CPWVKRFFA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.579676210 (376h45m37.350317545s) [noise=true , address=77.90.31.12:4615 , piece_id=GLIW3756SYGXSX44VFMMDNYVEJPS6O4ZBNS53OJRV67QNGMGLPUA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.595175322 (376h45m37.365816657s) [noise=true , address=77.90.31.16:4743 , piece_id=2VSOXKLN6Q434ET7JYLRSU6BW7PIKNZ43MZOQ4HQL6MDP3WUJOZA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.601273909 (376h45m37.371915244s) [noise=true , address=77.90.31.13:4643 , piece_id=PO24YVCOVVNORI7NRD3CDUC4ZPRULAXUESGBEDHJZYYDJ2ZUPQDA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:07.782538378 (376h45m42.553179714s) [noise=true , address=45.91.116.5:4322 , piece_id=QMOE7Z74V4HHQO7LIN6KAJHEUSDKIAVC4XLFH3AOJMC2ILP6I4FA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:22.662088184 (376h45m57.432729519s) [noise=true , address=104.36.82.17:4333 , piece_id=TPALHRSYWC3XKW77CPA36EZOOJHPLUEN4V6DWFN5NHLIAFSN5KRQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:24.053592013 (376h45m58.824233348s) [noise=true , address=104.36.82.2:4757 , piece_id=JMJIECO7WCFHWASHRCTEAG6VCU4S25SEETORWGJJQ63VAW533YJA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:44.900784468 (376h46m19.671425793s) [noise=true , address=104.36.82.23:4473 , piece_id=V3RAX7LZBU744EQQXDEG6Z25UUGQUFSSQCWCXNZKNGCTHD375GCA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:47.726372911 (376h46m22.497014236s) [noise=true , address=104.36.82.60:4364 , piece_id=7WZJU4UIO4FLPUNWUH2PVIIH3ZWWVZCP2D2UUSCNBCIS727VVWPA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:56.047624046 (376h46m30.818265381s) [noise=true , address=104.36.82.3:4791 , piece_id=NG7KRWBROBZWNP63A4734KWSRNGFVGLLXFZ5PGXZDJXFDTZNCZLQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:56.047644345 (376h46m30.81828568s) [seg_pos={0 0}, upload=301417]: Segment reached optimal threshold of 14 pieces.
Jul 9 01:56:56.047718136 (376h46m30.818359471s) [noise=true , address=104.36.82.38:4708 , piece_id=RDIWOSET3Q2JKHRQ7HVUSVMNILHVEDQ7XYDL5DMWK37Q7YTRADQA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BdJtXzvjMoNoQUjtHETyADPcZBTzcFzH8YDd651t7eTiFb1wf): protocol: expected piece hash; context canceled; EOF
Jul 9 01:56:56.047756840 (376h46m30.818398175s) [noise=true , address=104.36.82.22:4532 , piece_id=3BRLUKXPLFBXIPKURV3BBSTA562UWWQBZ75HNYXEX75E74FKJIGQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cHUuyc42zpw5k9Lz5Qo9AW9DMvKFPxRNdKZDZtjFaeyzT4Nkp): protocol: expected piece hash; context canceled; EOF
Jul 9 01:56:56.047948587 (376h46m30.818589922s) [noise=true , address=104.36.82.21:4508 , piece_id=VVX5U234EZZNCZWIVYJVYBFDYG2SCKWS5HT5GM4H7CNNJQXDHT3Q, seg_pos={0 0}, upload=301417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1269NRnqpabU6H8q9vSCmE8yMzznD1L7dyZFmzCWhtzJaTjbXg3): protocol: write tcp 188.166.225.170:60172->104.36.82.21:4508: use of closed network connection; write tcp 188.166.225.170:60172->104.36.82.21:4508: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60172->104.36.82.21:4508: use of closed network connection
Jul 9 01:56:56.047987552 (376h46m30.818628887s) [noise=true , address=104.36.82.6:4880 , piece_id=NKXAOUHYMQ4WR6V6ZGQOKRZWLAGI44AF4SNUR4OKUNJNICD4UUNQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125NsQhq7kRmTjKS8nuc51PZddApFEi5C38FzC5pEX2QrWYKzSW): protocol: write tcp 188.166.225.170:38944->104.36.82.6:4880: use of closed network connection; write tcp 188.166.225.170:38944->104.36.82.6:4880: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38944->104.36.82.6:4880: use of closed network connection
Jul 9 01:56:56.047996138 (376h46m30.818637463s) [seg_pos={0 0}, upload=301417]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 01:56:56.048046284 (376h46m30.818687619s) [upload=301417]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 01:56:56.048050091 (376h46m30.818691416s) [upload=301417]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 01:56:56.339173192 (376h46m31.109814527s) [upload=301417]: Done uploading segments...
Jul 9 01:56:56.339190416 (376h46m31.109831751s) [upload=301417]: Upload finished. err: <nil>
Duration: 42.669247452s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [...................X......X..........X.............X........] duration:
26.864975223s
FAIL (2025-07-09 02:02:39 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
hRequestItem_SegmentBegin
Jul 9 01:56:00.629444016 (376h45m35.400085361s) [seg_pos={0 0}, upload=301417]: Begin upload segment...
Jul 9 01:56:00.629512196 (376h45m35.400153531s) [seg_pos={0 0}, upload=301417]: Done begin upload segment.
Jul 9 01:56:00.629572552 (376h45m35.400213887s) [noise=true , address=104.36.82.22:4532 , piece_id=3BRLUKXPLFBXIPKURV3BBSTA562UWWQBZ75HNYXEX75E74FKJIGQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.629579284 (376h45m35.400220619s) [noise=true , address=77.90.31.10:4539 , piece_id=OI7V6HU5WMYZSQCS5PAN6OUCAJY4UGUZVTESMLB4LHOQJVMKJLHQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.629690497 (376h45m35.400331832s) [noise=true , address=104.36.82.17:4333 , piece_id=TPALHRSYWC3XKW77CPA36EZOOJHPLUEN4V6DWFN5NHLIAFSN5KRQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630082818 (376h45m35.400724153s) [noise=true , address=104.36.82.6:4880 , piece_id=NKXAOUHYMQ4WR6V6ZGQOKRZWLAGI44AF4SNUR4OKUNJNICD4UUNQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630158242 (376h45m35.400799577s) [noise=true , address=79.127.201.214:4308 , piece_id=V4LWWG3RM4JTLZGNZ3XJDHEOZO7XW5R7BHQQJ6Z4Y47PZCEZHN6A, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630428680 (376h45m35.401070025s) [noise=true , address=77.90.31.12:4615 , piece_id=GLIW3756SYGXSX44VFMMDNYVEJPS6O4ZBNS53OJRV67QNGMGLPUA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630538129 (376h45m35.401179464s) [noise=true , address=104.36.82.60:4364 , piece_id=7WZJU4UIO4FLPUNWUH2PVIIH3ZWWVZCP2D2UUSCNBCIS727VVWPA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630862820 (376h45m35.401504165s) [noise=true , address=104.36.82.38:4708 , piece_id=RDIWOSET3Q2JKHRQ7HVUSVMNILHVEDQ7XYDL5DMWK37Q7YTRADQA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.631210084 (376h45m35.401851419s) [noise=true , address=104.36.82.21:4508 , piece_id=VVX5U234EZZNCZWIVYJVYBFDYG2SCKWS5HT5GM4H7CNNJQXDHT3Q, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.631555495 (376h45m35.40219684s) [noise=true , address=77.90.31.16:4743 , piece_id=2VSOXKLN6Q434ET7JYLRSU6BW7PIKNZ43MZOQ4HQL6MDP3WUJOZA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.631989165 (376h45m35.4026305s) [noise=true , address=104.36.82.23:4473 , piece_id=V3RAX7LZBU744EQQXDEG6Z25UUGQUFSSQCWCXNZKNGCTHD375GCA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.632258450 (376h45m35.402899785s) [noise=true , address=77.90.31.19:4850 , piece_id=DNYDFQHORBVHQTPKFLWEGQF6QRLE6IIE6YGTRP6SS63CPWVKRFFA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.632521984 (376h45m35.403163309s) [noise=true , address=45.91.116.5:4322 , piece_id=QMOE7Z74V4HHQO7LIN6KAJHEUSDKIAVC4XLFH3AOJMC2ILP6I4FA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.632835304 (376h45m35.403476649s) [noise=true , address=104.36.82.3:4791 , piece_id=NG7KRWBROBZWNP63A4734KWSRNGFVGLLXFZ5PGXZDJXFDTZNCZLQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.633120529 (376h45m35.403761864s) [noise=true , address=77.90.31.14:4692 , piece_id=NHO2352W7MMVGRS3MNPWSEFHWBOK73SQ6JQJEWZ5TV5UDFY44H5Q, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.636119667 (376h45m35.406761002s) [noise=true , address=104.36.82.2:4757 , piece_id=JMJIECO7WCFHWASHRCTEAG6VCU4S25SEETORWGJJQ63VAW533YJA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.636833744 (376h45m35.407475079s) [noise=true , address=77.90.31.15:4712 , piece_id=3DN3NPUYRN4AIBYBV2UCGE4TTYXD2WJRLHDMQEFIU36HBTYDOJ5Q, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.637112587 (376h45m35.407753922s) [noise=true , address=77.90.31.13:4643 , piece_id=PO24YVCOVVNORI7NRD3CDUC4ZPRULAXUESGBEDHJZYYDJ2ZUPQDA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:02.137197163 (376h45m36.907838498s) [upload=301417]: Next returned nil segment
Jul 9 01:56:02.137214166 (376h45m36.907855501s) [upload=301417]: Waiting for error group managing segments...
Jul 9 01:56:02.521317983 (376h45m37.291959318s) [noise=true , address=79.127.201.214:4308 , piece_id=V4LWWG3RM4JTLZGNZ3XJDHEOZO7XW5R7BHQQJ6Z4Y47PZCEZHN6A, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.567664820 (376h45m37.338306156s) [noise=true , address=77.90.31.14:4692 , piece_id=NHO2352W7MMVGRS3MNPWSEFHWBOK73SQ6JQJEWZ5TV5UDFY44H5Q, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.568760968 (376h45m37.339402303s) [noise=true , address=77.90.31.10:4539 , piece_id=OI7V6HU5WMYZSQCS5PAN6OUCAJY4UGUZVTESMLB4LHOQJVMKJLHQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.573315612 (376h45m37.343956947s) [noise=true , address=77.90.31.15:4712 , piece_id=3DN3NPUYRN4AIBYBV2UCGE4TTYXD2WJRLHDMQEFIU36HBTYDOJ5Q, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.577761728 (376h45m37.348403053s) [noise=true , address=77.90.31.19:4850 , piece_id=DNYDFQHORBVHQTPKFLWEGQF6QRLE6IIE6YGTRP6SS63CPWVKRFFA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.579676210 (376h45m37.350317545s) [noise=true , address=77.90.31.12:4615 , piece_id=GLIW3756SYGXSX44VFMMDNYVEJPS6O4ZBNS53OJRV67QNGMGLPUA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.595175322 (376h45m37.365816657s) [noise=true , address=77.90.31.16:4743 , piece_id=2VSOXKLN6Q434ET7JYLRSU6BW7PIKNZ43MZOQ4HQL6MDP3WUJOZA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.601273909 (376h45m37.371915244s) [noise=true , address=77.90.31.13:4643 , piece_id=PO24YVCOVVNORI7NRD3CDUC4ZPRULAXUESGBEDHJZYYDJ2ZUPQDA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:07.782538378 (376h45m42.553179714s) [noise=true , address=45.91.116.5:4322 , piece_id=QMOE7Z74V4HHQO7LIN6KAJHEUSDKIAVC4XLFH3AOJMC2ILP6I4FA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:22.662088184 (376h45m57.432729519s) [noise=true , address=104.36.82.17:4333 , piece_id=TPALHRSYWC3XKW77CPA36EZOOJHPLUEN4V6DWFN5NHLIAFSN5KRQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:24.053592013 (376h45m58.824233348s) [noise=true , address=104.36.82.2:4757 , piece_id=JMJIECO7WCFHWASHRCTEAG6VCU4S25SEETORWGJJQ63VAW533YJA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:44.900784468 (376h46m19.671425793s) [noise=true , address=104.36.82.23:4473 , piece_id=V3RAX7LZBU744EQQXDEG6Z25UUGQUFSSQCWCXNZKNGCTHD375GCA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:47.726372911 (376h46m22.497014236s) [noise=true , address=104.36.82.60:4364 , piece_id=7WZJU4UIO4FLPUNWUH2PVIIH3ZWWVZCP2D2UUSCNBCIS727VVWPA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:56.047624046 (376h46m30.818265381s) [noise=true , address=104.36.82.3:4791 , piece_id=NG7KRWBROBZWNP63A4734KWSRNGFVGLLXFZ5PGXZDJXFDTZNCZLQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:56.047644345 (376h46m30.81828568s) [seg_pos={0 0}, upload=301417]: Segment reached optimal threshold of 14 pieces.
Jul 9 01:56:56.047718136 (376h46m30.818359471s) [noise=true , address=104.36.82.38:4708 , piece_id=RDIWOSET3Q2JKHRQ7HVUSVMNILHVEDQ7XYDL5DMWK37Q7YTRADQA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BdJtXzvjMoNoQUjtHETyADPcZBTzcFzH8YDd651t7eTiFb1wf): protocol: expected piece hash; context canceled; EOF
Jul 9 01:56:56.047756840 (376h46m30.818398175s) [noise=true , address=104.36.82.22:4532 , piece_id=3BRLUKXPLFBXIPKURV3BBSTA562UWWQBZ75HNYXEX75E74FKJIGQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cHUuyc42zpw5k9Lz5Qo9AW9DMvKFPxRNdKZDZtjFaeyzT4Nkp): protocol: expected piece hash; context canceled; EOF
Jul 9 01:56:56.047948587 (376h46m30.818589922s) [noise=true , address=104.36.82.21:4508 , piece_id=VVX5U234EZZNCZWIVYJVYBFDYG2SCKWS5HT5GM4H7CNNJQXDHT3Q, seg_pos={0 0}, upload=301417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1269NRnqpabU6H8q9vSCmE8yMzznD1L7dyZFmzCWhtzJaTjbXg3): protocol: write tcp 188.166.225.170:60172->104.36.82.21:4508: use of closed network connection; write tcp 188.166.225.170:60172->104.36.82.21:4508: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60172->104.36.82.21:4508: use of closed network connection
Jul 9 01:56:56.047987552 (376h46m30.818628887s) [noise=true , address=104.36.82.6:4880 , piece_id=NKXAOUHYMQ4WR6V6ZGQOKRZWLAGI44AF4SNUR4OKUNJNICD4UUNQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125NsQhq7kRmTjKS8nuc51PZddApFEi5C38FzC5pEX2QrWYKzSW): protocol: write tcp 188.166.225.170:38944->104.36.82.6:4880: use of closed network connection; write tcp 188.166.225.170:38944->104.36.82.6:4880: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38944->104.36.82.6:4880: use of closed network connection
Jul 9 01:56:56.047996138 (376h46m30.818637463s) [seg_pos={0 0}, upload=301417]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 01:56:56.048046284 (376h46m30.818687619s) [upload=301417]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 01:56:56.048050091 (376h46m30.818691416s) [upload=301417]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 01:56:56.339173192 (376h46m31.109814527s) [upload=301417]: Done uploading segments...
Jul 9 01:56:56.339190416 (376h46m31.109831751s) [upload=301417]: Upload finished. err: <nil>
Duration: 26.865000281s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [...X.X..X...X..X............X......X..X...X.X.X.X...X.......] duration:
6.288628222s
FAIL (2025-07-09 02:02:38 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.288648932s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [...........X......................X.................X......X] duration:
30.237354264s
FAIL (2025-07-09 02:02:30 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
nceled; EOF
Jul 9 02:00:49.060314467 (376h50m23.830955812s) [seg_pos={0 0}, upload=301481]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil>
Jul 9 02:00:49.060367628 (376h50m23.831008963s) [upload=301481]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 02:00:49.060371145 (376h50m23.83101248s) [upload=301481]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 02:00:49.357373945 (376h50m24.12801528s) [upload=301481]: Done uploading segments...
Jul 9 02:00:49.357395647 (376h50m24.128036982s) [upload=301481]: Upload finished. err: <nil>
Jul 9 02:02:01.314289870 (376h51m36.084931205s) [upload=301495]: Starting upload
Jul 9 02:02:01.314394990 (376h51m36.085036325s) [upload=301495]: Done starting upload
Jul 9 02:02:01.314484112 (376h51m36.085125457s) [upload=301495]: Uploading segments...
Jul 9 02:02:01.314699112 (376h51m36.085340457s) [upload=301495]: Got next segment. Inline: false
Jul 9 02:02:01.314747194 (376h51m36.085388529s) [upload=301495]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 02:02:01.314753927 (376h51m36.085395272s) [upload=301495]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 02:02:01.549200488 (376h51m36.319841873s) [seg_pos={0 0}, upload=301495]: Begin upload segment...
Jul 9 02:02:01.549266264 (376h51m36.319907609s) [seg_pos={0 0}, upload=301495]: Done begin upload segment.
Jul 9 02:02:01.549310128 (376h51m36.319951473s) [noise=true , address=104.36.82.7:4910 , piece_id=CPROZR2V6J7RKRZW6CNLA7P44DQVYH3VOZMYBBTEIBOL7MGINPDA, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.549336468 (376h51m36.319977823s) [noise=true , address=45.91.116.14:4190 , piece_id=ZAF4GOHABQQENBFRPE3HEMFTT474ULDEB5MZ5XDVVHFVTQID5AVA, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.549964509 (376h51m36.320605844s) [noise=true , address=104.36.82.13:4187 , piece_id=BLI6GHKTU3D5Q6FWTOS6UEPDLBDNCX2VEZCKAXIVAF32VWU7I6XQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.550253653 (376h51m36.320894988s) [noise=true , address=104.36.82.9:4006 , piece_id=XX7OUX5VBH5UVHELOP525HLHU5LA7O4B6DR6LKI7VYFWZYVAXHTQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.550518870 (376h51m36.321160225s) [noise=true , address=104.36.82.6:4874 , piece_id=UDKHL2V3C6ZYBR3FGHVIKQYJLPCOO3XLXYTSQJVLTGJ6I74IGGAA, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.550556472 (376h51m36.321197807s) [noise=true , address=45.91.116.10:4082 , piece_id=IRWOX4OQI5TB7FAT53RSX4C6DPNPUGZPPQXS6VFM2EE6FPZI4N4A, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.550837249 (376h51m36.321478584s) [noise=true , address=45.91.116.8:4048 , piece_id=UTXJQHFKWTANYVEEL2CICL5MQWVLUC4H4ZQ3IFXQD5O45RKB425A, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.551078060 (376h51m36.321719395s) [noise=true , address=45.91.116.7:4010 , piece_id=7CYJB5QPC3ZJZVGFNJKBZHOLSKPFBYMSWTJL37CQ73JIIM7LVA2A, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.551313831 (376h51m36.321955166s) [noise=true , address=77.90.31.11:4555 , piece_id=57OEP4BSHFGZQ5A7DJKJ4BNSO7ORPXPVMSN2I56YFREXE6N5RIPQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.551369758 (376h51m36.322011093s) [noise=true , address=45.91.116.11:4119 , piece_id=2TR3FC2ZYYNAPWL2GTL5EZVJW74NI5W275AYDNSZIME62TPILDBQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.551601652 (376h51m36.322242987s) [noise=true , address=104.36.82.5:4839 , piece_id=MM6CHXUZFEON5CG7GUKRLD7PGYKU3337UBU73DTHSKAKQUJUKYLQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.551803969 (376h51m36.322445304s) [noise=true , address=45.91.116.6:4973 , piece_id=2XDTRFYDWADB2U2DIT6VDWTHT6DVRPRNMCWJFW4Q7G7KXITLI6LQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.552051763 (376h51m36.322693098s) [noise=true , address=104.36.82.15:4258 , piece_id=KFL7XC6ILPLWJGYIGZVJ6X3ECS26XNNY6M4ZS3QHGXBPZG4XWILQ, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.552343200 (376h51m36.322984545s) [noise=true , address=45.91.116.13:4154 , piece_id=I2XLUYF36R6HKCEYHRUISRDMAPYP7CQHEXL7XB6IMG7GYO32W55Q, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.552474742 (376h51m36.323116077s) [noise=true , address=45.91.116.4:4939 , piece_id=QDXP222CYJXWF2NV5LKI7XOPMU7APMWDFQLWURFQG5BVJUNKNJBA, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:01.552718097 (376h51m36.323359433s) [noise=true , address=104.36.82.11:4114 , piece_id=R7MKGZKIKESIOB5CWCDO4KV2CNGN2EXTLGATSMK7Y2JMSFIB3DSA, seg_pos={0 0}, upload=301495]: Uploading piece...
Jul 9 02:02:03.251350348 (376h51m38.021991683s) [upload=301495]: Next returned nil segment
Jul 9 02:02:03.251367120 (376h51m38.022008455s) [upload=301495]: Waiting for error group managing segments...
Jul 9 02:02:03.536295419 (376h51m38.306936775s) [noise=true , address=104.36.82.9:4006 , piece_id=XX7OUX5VBH5UVHELOP525HLHU5LA7O4B6DR6LKI7VYFWZYVAXHTQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.585972469 (376h51m38.356613794s) [noise=true , address=45.91.116.8:4048 , piece_id=UTXJQHFKWTANYVEEL2CICL5MQWVLUC4H4ZQ3IFXQD5O45RKB425A, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.612355463 (376h51m38.382996798s) [noise=true , address=45.91.116.11:4119 , piece_id=2TR3FC2ZYYNAPWL2GTL5EZVJW74NI5W275AYDNSZIME62TPILDBQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.616140875 (376h51m38.38678221s) [noise=true , address=45.91.116.4:4939 , piece_id=QDXP222CYJXWF2NV5LKI7XOPMU7APMWDFQLWURFQG5BVJUNKNJBA, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.625063725 (376h51m38.395705061s) [noise=true , address=45.91.116.13:4154 , piece_id=I2XLUYF36R6HKCEYHRUISRDMAPYP7CQHEXL7XB6IMG7GYO32W55Q, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.627327436 (376h51m38.397968771s) [noise=true , address=77.90.31.11:4555 , piece_id=57OEP4BSHFGZQ5A7DJKJ4BNSO7ORPXPVMSN2I56YFREXE6N5RIPQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.680079459 (376h51m38.450720804s) [noise=true , address=45.91.116.14:4190 , piece_id=ZAF4GOHABQQENBFRPE3HEMFTT474ULDEB5MZ5XDVVHFVTQID5AVA, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.682275680 (376h51m38.452917015s) [noise=true , address=45.91.116.10:4082 , piece_id=IRWOX4OQI5TB7FAT53RSX4C6DPNPUGZPPQXS6VFM2EE6FPZI4N4A, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.693418348 (376h51m38.464059684s) [noise=true , address=45.91.116.6:4973 , piece_id=2XDTRFYDWADB2U2DIT6VDWTHT6DVRPRNMCWJFW4Q7G7KXITLI6LQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:03.714112589 (376h51m38.484753924s) [noise=true , address=45.91.116.7:4010 , piece_id=7CYJB5QPC3ZJZVGFNJKBZHOLSKPFBYMSWTJL37CQ73JIIM7LVA2A, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:24.743005877 (376h51m59.513647202s) [noise=true , address=104.36.82.6:4874 , piece_id=UDKHL2V3C6ZYBR3FGHVIKQYJLPCOO3XLXYTSQJVLTGJ6I74IGGAA, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:28.578275204 (376h52m3.348916539s) [noise=true , address=104.36.82.13:4187 , piece_id=BLI6GHKTU3D5Q6FWTOS6UEPDLBDNCX2VEZCKAXIVAF32VWU7I6XQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: <nil>
Jul 9 02:02:30.664723094 (376h52m5.435364439s) [noise=true , address=104.36.82.11:4114 , piece_id=R7MKGZKIKESIOB5CWCDO4KV2CNGN2EXTLGATSMK7Y2JMSFIB3DSA, seg_pos={0 0}, upload=301495]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 02:02:30.665014852 (376h52m5.435656188s) [noise=true , address=104.36.82.5:4839 , piece_id=MM6CHXUZFEON5CG7GUKRLD7PGYKU3337UBU73DTHSKAKQUJUKYLQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 188.166.225.170:42438->104.36.82.5:4839: use of closed network connection; write tcp 188.166.225.170:42438->104.36.82.5:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42438->104.36.82.5:4839: use of closed network connection
Jul 9 02:02:30.665113180 (376h52m5.435754525s) [noise=true , address=104.36.82.7:4910 , piece_id=CPROZR2V6J7RKRZW6CNLA7P44DQVYH3VOZMYBBTEIBOL7MGINPDA, seg_pos={0 0}, upload=301495]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: write tcp 188.166.225.170:38368->104.36.82.7:4910: use of closed network connection; write tcp 188.166.225.170:38368->104.36.82.7:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38368->104.36.82.7:4910: use of closed network connection
Jul 9 02:02:30.665314215 (376h52m5.43595555s) [noise=true , address=104.36.82.15:4258 , piece_id=KFL7XC6ILPLWJGYIGZVJ6X3ECS26XNNY6M4ZS3QHGXBPZG4XWILQ, seg_pos={0 0}, upload=301495]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 188.166.225.170:50202->104.36.82.15:4258: use of closed network connection; context deadline exceeded
Jul 9 02:02:30.665353560 (376h52m5.435994895s) [seg_pos={0 0}, upload=301495]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 02:02:30.900452118 (376h52m5.671093453s) [upload=301495]: Done uploading segments...
Jul 9 02:02:30.900486934 (376h52m5.67112827s) [upload=301495]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 30.237396876s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
16 out of 60 is failed (threshold=3) [........X......XX.XX..X......XXX....XX.....XX...X.XX........] duration:
37.321420614s
FAIL (2025-07-09 02:02:28 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 37.321435823s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [X...X.X..X...X..X............X......X..X...X.X.X.X...X......] duration:
15.640258786s
FAIL (2025-07-09 02:02:26 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 15.64028223s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [....................X......X..........X.............X.......] duration:
40.521977025s
FAIL (2025-07-09 02:01:57 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
hRequestItem_SegmentBegin
Jul 9 01:56:00.629444016 (376h45m35.400085361s) [seg_pos={0 0}, upload=301417]: Begin upload segment...
Jul 9 01:56:00.629512196 (376h45m35.400153531s) [seg_pos={0 0}, upload=301417]: Done begin upload segment.
Jul 9 01:56:00.629572552 (376h45m35.400213887s) [noise=true , address=104.36.82.22:4532 , piece_id=3BRLUKXPLFBXIPKURV3BBSTA562UWWQBZ75HNYXEX75E74FKJIGQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.629579284 (376h45m35.400220619s) [noise=true , address=77.90.31.10:4539 , piece_id=OI7V6HU5WMYZSQCS5PAN6OUCAJY4UGUZVTESMLB4LHOQJVMKJLHQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.629690497 (376h45m35.400331832s) [noise=true , address=104.36.82.17:4333 , piece_id=TPALHRSYWC3XKW77CPA36EZOOJHPLUEN4V6DWFN5NHLIAFSN5KRQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630082818 (376h45m35.400724153s) [noise=true , address=104.36.82.6:4880 , piece_id=NKXAOUHYMQ4WR6V6ZGQOKRZWLAGI44AF4SNUR4OKUNJNICD4UUNQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630158242 (376h45m35.400799577s) [noise=true , address=79.127.201.214:4308 , piece_id=V4LWWG3RM4JTLZGNZ3XJDHEOZO7XW5R7BHQQJ6Z4Y47PZCEZHN6A, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630428680 (376h45m35.401070025s) [noise=true , address=77.90.31.12:4615 , piece_id=GLIW3756SYGXSX44VFMMDNYVEJPS6O4ZBNS53OJRV67QNGMGLPUA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630538129 (376h45m35.401179464s) [noise=true , address=104.36.82.60:4364 , piece_id=7WZJU4UIO4FLPUNWUH2PVIIH3ZWWVZCP2D2UUSCNBCIS727VVWPA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630862820 (376h45m35.401504165s) [noise=true , address=104.36.82.38:4708 , piece_id=RDIWOSET3Q2JKHRQ7HVUSVMNILHVEDQ7XYDL5DMWK37Q7YTRADQA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.631210084 (376h45m35.401851419s) [noise=true , address=104.36.82.21:4508 , piece_id=VVX5U234EZZNCZWIVYJVYBFDYG2SCKWS5HT5GM4H7CNNJQXDHT3Q, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.631555495 (376h45m35.40219684s) [noise=true , address=77.90.31.16:4743 , piece_id=2VSOXKLN6Q434ET7JYLRSU6BW7PIKNZ43MZOQ4HQL6MDP3WUJOZA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.631989165 (376h45m35.4026305s) [noise=true , address=104.36.82.23:4473 , piece_id=V3RAX7LZBU744EQQXDEG6Z25UUGQUFSSQCWCXNZKNGCTHD375GCA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.632258450 (376h45m35.402899785s) [noise=true , address=77.90.31.19:4850 , piece_id=DNYDFQHORBVHQTPKFLWEGQF6QRLE6IIE6YGTRP6SS63CPWVKRFFA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.632521984 (376h45m35.403163309s) [noise=true , address=45.91.116.5:4322 , piece_id=QMOE7Z74V4HHQO7LIN6KAJHEUSDKIAVC4XLFH3AOJMC2ILP6I4FA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.632835304 (376h45m35.403476649s) [noise=true , address=104.36.82.3:4791 , piece_id=NG7KRWBROBZWNP63A4734KWSRNGFVGLLXFZ5PGXZDJXFDTZNCZLQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.633120529 (376h45m35.403761864s) [noise=true , address=77.90.31.14:4692 , piece_id=NHO2352W7MMVGRS3MNPWSEFHWBOK73SQ6JQJEWZ5TV5UDFY44H5Q, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.636119667 (376h45m35.406761002s) [noise=true , address=104.36.82.2:4757 , piece_id=JMJIECO7WCFHWASHRCTEAG6VCU4S25SEETORWGJJQ63VAW533YJA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.636833744 (376h45m35.407475079s) [noise=true , address=77.90.31.15:4712 , piece_id=3DN3NPUYRN4AIBYBV2UCGE4TTYXD2WJRLHDMQEFIU36HBTYDOJ5Q, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.637112587 (376h45m35.407753922s) [noise=true , address=77.90.31.13:4643 , piece_id=PO24YVCOVVNORI7NRD3CDUC4ZPRULAXUESGBEDHJZYYDJ2ZUPQDA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:02.137197163 (376h45m36.907838498s) [upload=301417]: Next returned nil segment
Jul 9 01:56:02.137214166 (376h45m36.907855501s) [upload=301417]: Waiting for error group managing segments...
Jul 9 01:56:02.521317983 (376h45m37.291959318s) [noise=true , address=79.127.201.214:4308 , piece_id=V4LWWG3RM4JTLZGNZ3XJDHEOZO7XW5R7BHQQJ6Z4Y47PZCEZHN6A, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.567664820 (376h45m37.338306156s) [noise=true , address=77.90.31.14:4692 , piece_id=NHO2352W7MMVGRS3MNPWSEFHWBOK73SQ6JQJEWZ5TV5UDFY44H5Q, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.568760968 (376h45m37.339402303s) [noise=true , address=77.90.31.10:4539 , piece_id=OI7V6HU5WMYZSQCS5PAN6OUCAJY4UGUZVTESMLB4LHOQJVMKJLHQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.573315612 (376h45m37.343956947s) [noise=true , address=77.90.31.15:4712 , piece_id=3DN3NPUYRN4AIBYBV2UCGE4TTYXD2WJRLHDMQEFIU36HBTYDOJ5Q, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.577761728 (376h45m37.348403053s) [noise=true , address=77.90.31.19:4850 , piece_id=DNYDFQHORBVHQTPKFLWEGQF6QRLE6IIE6YGTRP6SS63CPWVKRFFA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.579676210 (376h45m37.350317545s) [noise=true , address=77.90.31.12:4615 , piece_id=GLIW3756SYGXSX44VFMMDNYVEJPS6O4ZBNS53OJRV67QNGMGLPUA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.595175322 (376h45m37.365816657s) [noise=true , address=77.90.31.16:4743 , piece_id=2VSOXKLN6Q434ET7JYLRSU6BW7PIKNZ43MZOQ4HQL6MDP3WUJOZA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.601273909 (376h45m37.371915244s) [noise=true , address=77.90.31.13:4643 , piece_id=PO24YVCOVVNORI7NRD3CDUC4ZPRULAXUESGBEDHJZYYDJ2ZUPQDA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:07.782538378 (376h45m42.553179714s) [noise=true , address=45.91.116.5:4322 , piece_id=QMOE7Z74V4HHQO7LIN6KAJHEUSDKIAVC4XLFH3AOJMC2ILP6I4FA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:22.662088184 (376h45m57.432729519s) [noise=true , address=104.36.82.17:4333 , piece_id=TPALHRSYWC3XKW77CPA36EZOOJHPLUEN4V6DWFN5NHLIAFSN5KRQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:24.053592013 (376h45m58.824233348s) [noise=true , address=104.36.82.2:4757 , piece_id=JMJIECO7WCFHWASHRCTEAG6VCU4S25SEETORWGJJQ63VAW533YJA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:44.900784468 (376h46m19.671425793s) [noise=true , address=104.36.82.23:4473 , piece_id=V3RAX7LZBU744EQQXDEG6Z25UUGQUFSSQCWCXNZKNGCTHD375GCA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:47.726372911 (376h46m22.497014236s) [noise=true , address=104.36.82.60:4364 , piece_id=7WZJU4UIO4FLPUNWUH2PVIIH3ZWWVZCP2D2UUSCNBCIS727VVWPA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:56.047624046 (376h46m30.818265381s) [noise=true , address=104.36.82.3:4791 , piece_id=NG7KRWBROBZWNP63A4734KWSRNGFVGLLXFZ5PGXZDJXFDTZNCZLQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:56.047644345 (376h46m30.81828568s) [seg_pos={0 0}, upload=301417]: Segment reached optimal threshold of 14 pieces.
Jul 9 01:56:56.047718136 (376h46m30.818359471s) [noise=true , address=104.36.82.38:4708 , piece_id=RDIWOSET3Q2JKHRQ7HVUSVMNILHVEDQ7XYDL5DMWK37Q7YTRADQA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BdJtXzvjMoNoQUjtHETyADPcZBTzcFzH8YDd651t7eTiFb1wf): protocol: expected piece hash; context canceled; EOF
Jul 9 01:56:56.047756840 (376h46m30.818398175s) [noise=true , address=104.36.82.22:4532 , piece_id=3BRLUKXPLFBXIPKURV3BBSTA562UWWQBZ75HNYXEX75E74FKJIGQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cHUuyc42zpw5k9Lz5Qo9AW9DMvKFPxRNdKZDZtjFaeyzT4Nkp): protocol: expected piece hash; context canceled; EOF
Jul 9 01:56:56.047948587 (376h46m30.818589922s) [noise=true , address=104.36.82.21:4508 , piece_id=VVX5U234EZZNCZWIVYJVYBFDYG2SCKWS5HT5GM4H7CNNJQXDHT3Q, seg_pos={0 0}, upload=301417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1269NRnqpabU6H8q9vSCmE8yMzznD1L7dyZFmzCWhtzJaTjbXg3): protocol: write tcp 188.166.225.170:60172->104.36.82.21:4508: use of closed network connection; write tcp 188.166.225.170:60172->104.36.82.21:4508: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60172->104.36.82.21:4508: use of closed network connection
Jul 9 01:56:56.047987552 (376h46m30.818628887s) [noise=true , address=104.36.82.6:4880 , piece_id=NKXAOUHYMQ4WR6V6ZGQOKRZWLAGI44AF4SNUR4OKUNJNICD4UUNQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125NsQhq7kRmTjKS8nuc51PZddApFEi5C38FzC5pEX2QrWYKzSW): protocol: write tcp 188.166.225.170:38944->104.36.82.6:4880: use of closed network connection; write tcp 188.166.225.170:38944->104.36.82.6:4880: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38944->104.36.82.6:4880: use of closed network connection
Jul 9 01:56:56.047996138 (376h46m30.818637463s) [seg_pos={0 0}, upload=301417]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 01:56:56.048046284 (376h46m30.818687619s) [upload=301417]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 01:56:56.048050091 (376h46m30.818691416s) [upload=301417]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 01:56:56.339173192 (376h46m31.109814527s) [upload=301417]: Done uploading segments...
Jul 9 01:56:56.339190416 (376h46m31.109831751s) [upload=301417]: Upload finished. err: <nil>
Duration: 40.521992795s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
16 out of 60 is failed (threshold=3) [.........X......XX.XX..X......XXX....XX.....XX...X.XX.......] duration:
28.905578241s
FAIL (2025-07-09 02:01:51 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 28.90559886s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [.X...X.X..X...X..X............X......X..X...X.X.X.X...X.....] duration:
12.096052929s
FAIL (2025-07-09 02:01:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 12.096075362s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
16 out of 60 is failed (threshold=3) [..........X......XX.XX..X......XXX....XX.....XX...X.XX......] duration:
51.430130381s
FAIL (2025-07-09 02:01:21 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 51.430142655s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [.....................X......X..........X.............X......] duration:
33.49603374s
FAIL (2025-07-09 02:01:16 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
hRequestItem_SegmentBegin
Jul 9 01:56:00.629444016 (376h45m35.400085361s) [seg_pos={0 0}, upload=301417]: Begin upload segment...
Jul 9 01:56:00.629512196 (376h45m35.400153531s) [seg_pos={0 0}, upload=301417]: Done begin upload segment.
Jul 9 01:56:00.629572552 (376h45m35.400213887s) [noise=true , address=104.36.82.22:4532 , piece_id=3BRLUKXPLFBXIPKURV3BBSTA562UWWQBZ75HNYXEX75E74FKJIGQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.629579284 (376h45m35.400220619s) [noise=true , address=77.90.31.10:4539 , piece_id=OI7V6HU5WMYZSQCS5PAN6OUCAJY4UGUZVTESMLB4LHOQJVMKJLHQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.629690497 (376h45m35.400331832s) [noise=true , address=104.36.82.17:4333 , piece_id=TPALHRSYWC3XKW77CPA36EZOOJHPLUEN4V6DWFN5NHLIAFSN5KRQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630082818 (376h45m35.400724153s) [noise=true , address=104.36.82.6:4880 , piece_id=NKXAOUHYMQ4WR6V6ZGQOKRZWLAGI44AF4SNUR4OKUNJNICD4UUNQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630158242 (376h45m35.400799577s) [noise=true , address=79.127.201.214:4308 , piece_id=V4LWWG3RM4JTLZGNZ3XJDHEOZO7XW5R7BHQQJ6Z4Y47PZCEZHN6A, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630428680 (376h45m35.401070025s) [noise=true , address=77.90.31.12:4615 , piece_id=GLIW3756SYGXSX44VFMMDNYVEJPS6O4ZBNS53OJRV67QNGMGLPUA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630538129 (376h45m35.401179464s) [noise=true , address=104.36.82.60:4364 , piece_id=7WZJU4UIO4FLPUNWUH2PVIIH3ZWWVZCP2D2UUSCNBCIS727VVWPA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630862820 (376h45m35.401504165s) [noise=true , address=104.36.82.38:4708 , piece_id=RDIWOSET3Q2JKHRQ7HVUSVMNILHVEDQ7XYDL5DMWK37Q7YTRADQA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.631210084 (376h45m35.401851419s) [noise=true , address=104.36.82.21:4508 , piece_id=VVX5U234EZZNCZWIVYJVYBFDYG2SCKWS5HT5GM4H7CNNJQXDHT3Q, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.631555495 (376h45m35.40219684s) [noise=true , address=77.90.31.16:4743 , piece_id=2VSOXKLN6Q434ET7JYLRSU6BW7PIKNZ43MZOQ4HQL6MDP3WUJOZA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.631989165 (376h45m35.4026305s) [noise=true , address=104.36.82.23:4473 , piece_id=V3RAX7LZBU744EQQXDEG6Z25UUGQUFSSQCWCXNZKNGCTHD375GCA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.632258450 (376h45m35.402899785s) [noise=true , address=77.90.31.19:4850 , piece_id=DNYDFQHORBVHQTPKFLWEGQF6QRLE6IIE6YGTRP6SS63CPWVKRFFA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.632521984 (376h45m35.403163309s) [noise=true , address=45.91.116.5:4322 , piece_id=QMOE7Z74V4HHQO7LIN6KAJHEUSDKIAVC4XLFH3AOJMC2ILP6I4FA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.632835304 (376h45m35.403476649s) [noise=true , address=104.36.82.3:4791 , piece_id=NG7KRWBROBZWNP63A4734KWSRNGFVGLLXFZ5PGXZDJXFDTZNCZLQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.633120529 (376h45m35.403761864s) [noise=true , address=77.90.31.14:4692 , piece_id=NHO2352W7MMVGRS3MNPWSEFHWBOK73SQ6JQJEWZ5TV5UDFY44H5Q, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.636119667 (376h45m35.406761002s) [noise=true , address=104.36.82.2:4757 , piece_id=JMJIECO7WCFHWASHRCTEAG6VCU4S25SEETORWGJJQ63VAW533YJA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.636833744 (376h45m35.407475079s) [noise=true , address=77.90.31.15:4712 , piece_id=3DN3NPUYRN4AIBYBV2UCGE4TTYXD2WJRLHDMQEFIU36HBTYDOJ5Q, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.637112587 (376h45m35.407753922s) [noise=true , address=77.90.31.13:4643 , piece_id=PO24YVCOVVNORI7NRD3CDUC4ZPRULAXUESGBEDHJZYYDJ2ZUPQDA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:02.137197163 (376h45m36.907838498s) [upload=301417]: Next returned nil segment
Jul 9 01:56:02.137214166 (376h45m36.907855501s) [upload=301417]: Waiting for error group managing segments...
Jul 9 01:56:02.521317983 (376h45m37.291959318s) [noise=true , address=79.127.201.214:4308 , piece_id=V4LWWG3RM4JTLZGNZ3XJDHEOZO7XW5R7BHQQJ6Z4Y47PZCEZHN6A, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.567664820 (376h45m37.338306156s) [noise=true , address=77.90.31.14:4692 , piece_id=NHO2352W7MMVGRS3MNPWSEFHWBOK73SQ6JQJEWZ5TV5UDFY44H5Q, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.568760968 (376h45m37.339402303s) [noise=true , address=77.90.31.10:4539 , piece_id=OI7V6HU5WMYZSQCS5PAN6OUCAJY4UGUZVTESMLB4LHOQJVMKJLHQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.573315612 (376h45m37.343956947s) [noise=true , address=77.90.31.15:4712 , piece_id=3DN3NPUYRN4AIBYBV2UCGE4TTYXD2WJRLHDMQEFIU36HBTYDOJ5Q, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.577761728 (376h45m37.348403053s) [noise=true , address=77.90.31.19:4850 , piece_id=DNYDFQHORBVHQTPKFLWEGQF6QRLE6IIE6YGTRP6SS63CPWVKRFFA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.579676210 (376h45m37.350317545s) [noise=true , address=77.90.31.12:4615 , piece_id=GLIW3756SYGXSX44VFMMDNYVEJPS6O4ZBNS53OJRV67QNGMGLPUA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.595175322 (376h45m37.365816657s) [noise=true , address=77.90.31.16:4743 , piece_id=2VSOXKLN6Q434ET7JYLRSU6BW7PIKNZ43MZOQ4HQL6MDP3WUJOZA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.601273909 (376h45m37.371915244s) [noise=true , address=77.90.31.13:4643 , piece_id=PO24YVCOVVNORI7NRD3CDUC4ZPRULAXUESGBEDHJZYYDJ2ZUPQDA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:07.782538378 (376h45m42.553179714s) [noise=true , address=45.91.116.5:4322 , piece_id=QMOE7Z74V4HHQO7LIN6KAJHEUSDKIAVC4XLFH3AOJMC2ILP6I4FA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:22.662088184 (376h45m57.432729519s) [noise=true , address=104.36.82.17:4333 , piece_id=TPALHRSYWC3XKW77CPA36EZOOJHPLUEN4V6DWFN5NHLIAFSN5KRQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:24.053592013 (376h45m58.824233348s) [noise=true , address=104.36.82.2:4757 , piece_id=JMJIECO7WCFHWASHRCTEAG6VCU4S25SEETORWGJJQ63VAW533YJA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:44.900784468 (376h46m19.671425793s) [noise=true , address=104.36.82.23:4473 , piece_id=V3RAX7LZBU744EQQXDEG6Z25UUGQUFSSQCWCXNZKNGCTHD375GCA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:47.726372911 (376h46m22.497014236s) [noise=true , address=104.36.82.60:4364 , piece_id=7WZJU4UIO4FLPUNWUH2PVIIH3ZWWVZCP2D2UUSCNBCIS727VVWPA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:56.047624046 (376h46m30.818265381s) [noise=true , address=104.36.82.3:4791 , piece_id=NG7KRWBROBZWNP63A4734KWSRNGFVGLLXFZ5PGXZDJXFDTZNCZLQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:56.047644345 (376h46m30.81828568s) [seg_pos={0 0}, upload=301417]: Segment reached optimal threshold of 14 pieces.
Jul 9 01:56:56.047718136 (376h46m30.818359471s) [noise=true , address=104.36.82.38:4708 , piece_id=RDIWOSET3Q2JKHRQ7HVUSVMNILHVEDQ7XYDL5DMWK37Q7YTRADQA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BdJtXzvjMoNoQUjtHETyADPcZBTzcFzH8YDd651t7eTiFb1wf): protocol: expected piece hash; context canceled; EOF
Jul 9 01:56:56.047756840 (376h46m30.818398175s) [noise=true , address=104.36.82.22:4532 , piece_id=3BRLUKXPLFBXIPKURV3BBSTA562UWWQBZ75HNYXEX75E74FKJIGQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cHUuyc42zpw5k9Lz5Qo9AW9DMvKFPxRNdKZDZtjFaeyzT4Nkp): protocol: expected piece hash; context canceled; EOF
Jul 9 01:56:56.047948587 (376h46m30.818589922s) [noise=true , address=104.36.82.21:4508 , piece_id=VVX5U234EZZNCZWIVYJVYBFDYG2SCKWS5HT5GM4H7CNNJQXDHT3Q, seg_pos={0 0}, upload=301417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1269NRnqpabU6H8q9vSCmE8yMzznD1L7dyZFmzCWhtzJaTjbXg3): protocol: write tcp 188.166.225.170:60172->104.36.82.21:4508: use of closed network connection; write tcp 188.166.225.170:60172->104.36.82.21:4508: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60172->104.36.82.21:4508: use of closed network connection
Jul 9 01:56:56.047987552 (376h46m30.818628887s) [noise=true , address=104.36.82.6:4880 , piece_id=NKXAOUHYMQ4WR6V6ZGQOKRZWLAGI44AF4SNUR4OKUNJNICD4UUNQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125NsQhq7kRmTjKS8nuc51PZddApFEi5C38FzC5pEX2QrWYKzSW): protocol: write tcp 188.166.225.170:38944->104.36.82.6:4880: use of closed network connection; write tcp 188.166.225.170:38944->104.36.82.6:4880: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38944->104.36.82.6:4880: use of closed network connection
Jul 9 01:56:56.047996138 (376h46m30.818637463s) [seg_pos={0 0}, upload=301417]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 01:56:56.048046284 (376h46m30.818687619s) [upload=301417]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 01:56:56.048050091 (376h46m30.818691416s) [upload=301417]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 01:56:56.339173192 (376h46m31.109814527s) [upload=301417]: Done uploading segments...
Jul 9 01:56:56.339190416 (376h46m31.109831751s) [upload=301417]: Upload finished. err: <nil>
Duration: 33.49605973s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [..X...X.X..X...X..X............X......X..X...X.X.X.X...X....] duration:
10.465504106s
FAIL (2025-07-09 02:01:15 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 10.46552697s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [............X......................X.................X......] duration:
5.685829542s
FAIL (2025-07-09 02:00:51 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
em: *pb.BatchRequestItem_SegmentCommit
Jul 9 01:55:26.378238935 (376h45m1.14888026s) [upload=301404]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 01:55:26.681901575 (376h45m1.45254291s) [upload=301404]: Done uploading segments...
Jul 9 01:55:26.681918448 (376h45m1.452559783s) [upload=301404]: Upload finished. err: <nil>
Jul 9 01:56:07.557068843 (376h45m42.327710178s) [upload=301419]: Starting upload
Jul 9 01:56:07.557123908 (376h45m42.327765243s) [upload=301419]: Done starting upload
Jul 9 01:56:07.557294845 (376h45m42.32793618s) [upload=301419]: Uploading segments...
Jul 9 01:56:07.557334942 (376h45m42.327976267s) [upload=301419]: Got next segment. Inline: false
Jul 9 01:56:07.557358848 (376h45m42.328000183s) [upload=301419]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 01:56:07.557363737 (376h45m42.328005072s) [upload=301419]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 01:56:07.786768822 (376h45m42.557410157s) [seg_pos={0 0}, upload=301419]: Begin upload segment...
Jul 9 01:56:07.786807697 (376h45m42.557449032s) [seg_pos={0 0}, upload=301419]: Done begin upload segment.
Jul 9 01:56:07.786827744 (376h45m42.557469069s) [noise=true , address=45.91.116.13:4154 , piece_id=WT7WL2P7RMJFA7XIZ3KNRVJ2RWDQGQBCM3J65X2X3MQ6WSAKHOIQ, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.786867240 (376h45m42.557508576s) [noise=true , address=104.36.82.2:4732 , piece_id=HF35AWQTHK4B5XQXLUOM5HPEBINSUIXGA3AAMZFEHZW4GDRDT4KQ, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.787161643 (376h45m42.557802968s) [noise=true , address=104.36.82.16:4294 , piece_id=TWDA4QZ5WAQF3OOXDDNAHBKYM5OAKBZ34QWW2FYXHJ6KGKDTDFRQ, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.787243289 (376h45m42.557884614s) [noise=true , address=77.90.31.11:4555 , piece_id=A7UGOPRF44445XR4OBEODVMMJU4HED7JUOPORHJKXVEPJO6KYIWQ, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.787405490 (376h45m42.558046815s) [noise=true , address=77.90.31.10:4518 , piece_id=ZC7HW4EDPE6UU2MLOHGHMYKBA4NXTJEL4I6CP6FK62QIAOEUT4ZQ, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.787501334 (376h45m42.558142669s) [noise=true , address=45.91.116.6:4975 , piece_id=MRETRUDTUOXXPHG4P25RMAJP4EDX4SS4B7O22PPPIQBSULZ52KNA, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.787663624 (376h45m42.558304959s) [noise=true , address=104.36.82.4:4801 , piece_id=OVVFQSK4TYRAXFNR6NZ74463CDC5BPKVROPU45ATGOTCZDUFDWHQ, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.787980670 (376h45m42.558622005s) [noise=true , address=45.91.116.11:4117 , piece_id=VDNXPWCQBXOOEAEJB7YB75AWP25A2AMCLTOT6KHFETFUXZ6UH3IQ, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.788245387 (376h45m42.558886712s) [noise=true , address=104.36.82.10:4078 , piece_id=ORPJSLOMY4IAB2NWYOQ7RCZCLEIQ25CIXFKKTWW5LBO5BFPPY64Q, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.788473754 (376h45m42.559115079s) [noise=true , address=45.91.116.7:4009 , piece_id=EEGGWRYW7EDJOVZDWO36KAGPFBQFSK5FJ7YU2FPBBWSIGV2KM4UA, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.788656052 (376h45m42.559297387s) [noise=true , address=104.36.82.6:4876 , piece_id=TUZF2S3DJSUK6MMNEQYWR35I56NAFLDRFPL646TIAGIUNVWE2ZGA, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.788848981 (376h45m42.559490316s) [noise=true , address=45.91.116.10:4081 , piece_id=4ZKRZFKWSILUJ3QNHIDRBGUETYMO7UZI5G5XR64XI2ANHXRCKAZQ, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.789032262 (376h45m42.559673597s) [noise=true , address=104.36.82.9:4006 , piece_id=P7YEMWPREVLDDFE7JDYAOEZJIYCAL3OAKKJJDC6EBMLLYSVRL73Q, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.789272181 (376h45m42.559913506s) [noise=true , address=77.90.31.19:4843 , piece_id=5FRTA2EJNZVNHV47EWYRFUH6VW2FJN2EBAKBRJWMP66BXF6SZF6A, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.789459429 (376h45m42.560100764s) [noise=true , address=45.91.116.8:4047 , piece_id=VISW3TARCTQMHB3662HBA2I7UW4IIREOJ32XOSBNTRPPTB4KD77A, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.789640225 (376h45m42.56028155s) [noise=true , address=77.90.31.21:4877 , piece_id=EN7QS3CEXXIBY7FU6Y32LLDKYCSVRJQ4G7RXRE2DV7CUPMZBVHDQ, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:09.422697440 (376h45m44.193338775s) [upload=301419]: Next returned nil segment
Jul 9 01:56:09.422714812 (376h45m44.193356148s) [upload=301419]: Waiting for error group managing segments...
Jul 9 01:56:09.703007945 (376h45m44.47364928s) [noise=true , address=104.36.82.9:4006 , piece_id=P7YEMWPREVLDDFE7JDYAOEZJIYCAL3OAKKJJDC6EBMLLYSVRL73Q, seg_pos={0 0}, upload=301419]: Done uploading piece. err: <nil>
Jul 9 01:56:09.780867601 (376h45m44.551508936s) [noise=true , address=104.36.82.16:4294 , piece_id=TWDA4QZ5WAQF3OOXDDNAHBKYM5OAKBZ34QWW2FYXHJ6KGKDTDFRQ, seg_pos={0 0}, upload=301419]: Done uploading piece. err: <nil>
Jul 9 01:56:09.840231843 (376h45m44.610873178s) [noise=true , address=45.91.116.6:4975 , piece_id=MRETRUDTUOXXPHG4P25RMAJP4EDX4SS4B7O22PPPIQBSULZ52KNA, seg_pos={0 0}, upload=301419]: Done uploading piece. err: <nil>
Jul 9 01:56:09.841984998 (376h45m44.612626363s) [noise=true , address=45.91.116.10:4081 , piece_id=4ZKRZFKWSILUJ3QNHIDRBGUETYMO7UZI5G5XR64XI2ANHXRCKAZQ, seg_pos={0 0}, upload=301419]: Done uploading piece. err: <nil>
Jul 9 01:56:09.850995516 (376h45m44.621636851s) [noise=true , address=45.91.116.13:4154 , piece_id=WT7WL2P7RMJFA7XIZ3KNRVJ2RWDQGQBCM3J65X2X3MQ6WSAKHOIQ, seg_pos={0 0}, upload=301419]: Done uploading piece. err: <nil>
Jul 9 01:56:09.860567239 (376h45m44.631208574s) [noise=true , address=45.91.116.7:4009 , piece_id=EEGGWRYW7EDJOVZDWO36KAGPFBQFSK5FJ7YU2FPBBWSIGV2KM4UA, seg_pos={0 0}, upload=301419]: Done uploading piece. err: <nil>
Jul 9 01:56:09.871129736 (376h45m44.641771071s) [noise=true , address=77.90.31.19:4843 , piece_id=5FRTA2EJNZVNHV47EWYRFUH6VW2FJN2EBAKBRJWMP66BXF6SZF6A, seg_pos={0 0}, upload=301419]: Done uploading piece. err: <nil>
Jul 9 01:56:09.874489343 (376h45m44.645130678s) [noise=true , address=77.90.31.11:4555 , piece_id=A7UGOPRF44445XR4OBEODVMMJU4HED7JUOPORHJKXVEPJO6KYIWQ, seg_pos={0 0}, upload=301419]: Done uploading piece. err: <nil>
Jul 9 01:56:09.875282992 (376h45m44.645924327s) [noise=true , address=77.90.31.21:4877 , piece_id=EN7QS3CEXXIBY7FU6Y32LLDKYCSVRJQ4G7RXRE2DV7CUPMZBVHDQ, seg_pos={0 0}, upload=301419]: Done uploading piece. err: <nil>
Jul 9 01:56:09.882277543 (376h45m44.652918878s) [noise=true , address=77.90.31.10:4518 , piece_id=ZC7HW4EDPE6UU2MLOHGHMYKBA4NXTJEL4I6CP6FK62QIAOEUT4ZQ, seg_pos={0 0}, upload=301419]: Done uploading piece. err: <nil>
Jul 9 01:56:09.909865152 (376h45m44.680506487s) [noise=true , address=45.91.116.11:4117 , piece_id=VDNXPWCQBXOOEAEJB7YB75AWP25A2AMCLTOT6KHFETFUXZ6UH3IQ, seg_pos={0 0}, upload=301419]: Done uploading piece. err: <nil>
Jul 9 01:56:09.930049268 (376h45m44.700690603s) [noise=true , address=45.91.116.8:4047 , piece_id=VISW3TARCTQMHB3662HBA2I7UW4IIREOJ32XOSBNTRPPTB4KD77A, seg_pos={0 0}, upload=301419]: Done uploading piece. err: <nil>
Jul 9 01:56:36.916221275 (376h46m11.68686263s) [noise=true , address=104.36.82.6:4876 , piece_id=TUZF2S3DJSUK6MMNEQYWR35I56NAFLDRFPL646TIAGIUNVWE2ZGA, seg_pos={0 0}, upload=301419]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 188.166.225.170:36068->104.36.82.6:4876: use of closed network connection; write tcp 188.166.225.170:36068->104.36.82.6:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36068->104.36.82.6:4876: use of closed network connection
Jul 9 01:56:36.916318731 (376h46m11.686960076s) [noise=true , address=104.36.82.4:4801 , piece_id=OVVFQSK4TYRAXFNR6NZ74463CDC5BPKVROPU45ATGOTCZDUFDWHQ, seg_pos={0 0}, upload=301419]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 188.166.225.170:36588->104.36.82.4:4801: use of closed network connection; context deadline exceeded
Jul 9 01:56:36.916361463 (376h46m11.687002818s) [noise=true , address=104.36.82.2:4732 , piece_id=HF35AWQTHK4B5XQXLUOM5HPEBINSUIXGA3AAMZFEHZW4GDRDT4KQ, seg_pos={0 0}, upload=301419]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: write tcp 188.166.225.170:44126->104.36.82.2:4732: use of closed network connection; write tcp 188.166.225.170:44126->104.36.82.2:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44126->104.36.82.2:4732: use of closed network connection
Jul 9 01:56:36.916428551 (376h46m11.687069916s) [noise=true , address=104.36.82.10:4078 , piece_id=ORPJSLOMY4IAB2NWYOQ7RCZCLEIQ25CIXFKKTWW5LBO5BFPPY64Q, seg_pos={0 0}, upload=301419]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 188.166.225.170:53862->104.36.82.10:4078: use of closed network connection; write tcp 188.166.225.170:53862->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53862->104.36.82.10:4078: use of closed network connection
Jul 9 01:56:36.916470932 (376h46m11.687112338s) [seg_pos={0 0}, upload=301419]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 01:56:37.151140589 (376h46m11.921781964s) [upload=301419]: Done uploading segments...
Jul 9 01:56:37.151169034 (376h46m11.921810369s) [upload=301419]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.685853868s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [...X...X.X..X...X..X............X......X..X...X.X.X.X...X...] duration:
11.066472509s
FAIL (2025-07-09 02:00:42 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 11.066494731s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [......................X......X..........X.............X.....] duration:
37.223254173s
FAIL (2025-07-09 02:00:37 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
hRequestItem_SegmentBegin
Jul 9 01:56:00.629444016 (376h45m35.400085361s) [seg_pos={0 0}, upload=301417]: Begin upload segment...
Jul 9 01:56:00.629512196 (376h45m35.400153531s) [seg_pos={0 0}, upload=301417]: Done begin upload segment.
Jul 9 01:56:00.629572552 (376h45m35.400213887s) [noise=true , address=104.36.82.22:4532 , piece_id=3BRLUKXPLFBXIPKURV3BBSTA562UWWQBZ75HNYXEX75E74FKJIGQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.629579284 (376h45m35.400220619s) [noise=true , address=77.90.31.10:4539 , piece_id=OI7V6HU5WMYZSQCS5PAN6OUCAJY4UGUZVTESMLB4LHOQJVMKJLHQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.629690497 (376h45m35.400331832s) [noise=true , address=104.36.82.17:4333 , piece_id=TPALHRSYWC3XKW77CPA36EZOOJHPLUEN4V6DWFN5NHLIAFSN5KRQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630082818 (376h45m35.400724153s) [noise=true , address=104.36.82.6:4880 , piece_id=NKXAOUHYMQ4WR6V6ZGQOKRZWLAGI44AF4SNUR4OKUNJNICD4UUNQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630158242 (376h45m35.400799577s) [noise=true , address=79.127.201.214:4308 , piece_id=V4LWWG3RM4JTLZGNZ3XJDHEOZO7XW5R7BHQQJ6Z4Y47PZCEZHN6A, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630428680 (376h45m35.401070025s) [noise=true , address=77.90.31.12:4615 , piece_id=GLIW3756SYGXSX44VFMMDNYVEJPS6O4ZBNS53OJRV67QNGMGLPUA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630538129 (376h45m35.401179464s) [noise=true , address=104.36.82.60:4364 , piece_id=7WZJU4UIO4FLPUNWUH2PVIIH3ZWWVZCP2D2UUSCNBCIS727VVWPA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630862820 (376h45m35.401504165s) [noise=true , address=104.36.82.38:4708 , piece_id=RDIWOSET3Q2JKHRQ7HVUSVMNILHVEDQ7XYDL5DMWK37Q7YTRADQA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.631210084 (376h45m35.401851419s) [noise=true , address=104.36.82.21:4508 , piece_id=VVX5U234EZZNCZWIVYJVYBFDYG2SCKWS5HT5GM4H7CNNJQXDHT3Q, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.631555495 (376h45m35.40219684s) [noise=true , address=77.90.31.16:4743 , piece_id=2VSOXKLN6Q434ET7JYLRSU6BW7PIKNZ43MZOQ4HQL6MDP3WUJOZA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.631989165 (376h45m35.4026305s) [noise=true , address=104.36.82.23:4473 , piece_id=V3RAX7LZBU744EQQXDEG6Z25UUGQUFSSQCWCXNZKNGCTHD375GCA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.632258450 (376h45m35.402899785s) [noise=true , address=77.90.31.19:4850 , piece_id=DNYDFQHORBVHQTPKFLWEGQF6QRLE6IIE6YGTRP6SS63CPWVKRFFA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.632521984 (376h45m35.403163309s) [noise=true , address=45.91.116.5:4322 , piece_id=QMOE7Z74V4HHQO7LIN6KAJHEUSDKIAVC4XLFH3AOJMC2ILP6I4FA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.632835304 (376h45m35.403476649s) [noise=true , address=104.36.82.3:4791 , piece_id=NG7KRWBROBZWNP63A4734KWSRNGFVGLLXFZ5PGXZDJXFDTZNCZLQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.633120529 (376h45m35.403761864s) [noise=true , address=77.90.31.14:4692 , piece_id=NHO2352W7MMVGRS3MNPWSEFHWBOK73SQ6JQJEWZ5TV5UDFY44H5Q, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.636119667 (376h45m35.406761002s) [noise=true , address=104.36.82.2:4757 , piece_id=JMJIECO7WCFHWASHRCTEAG6VCU4S25SEETORWGJJQ63VAW533YJA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.636833744 (376h45m35.407475079s) [noise=true , address=77.90.31.15:4712 , piece_id=3DN3NPUYRN4AIBYBV2UCGE4TTYXD2WJRLHDMQEFIU36HBTYDOJ5Q, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.637112587 (376h45m35.407753922s) [noise=true , address=77.90.31.13:4643 , piece_id=PO24YVCOVVNORI7NRD3CDUC4ZPRULAXUESGBEDHJZYYDJ2ZUPQDA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:02.137197163 (376h45m36.907838498s) [upload=301417]: Next returned nil segment
Jul 9 01:56:02.137214166 (376h45m36.907855501s) [upload=301417]: Waiting for error group managing segments...
Jul 9 01:56:02.521317983 (376h45m37.291959318s) [noise=true , address=79.127.201.214:4308 , piece_id=V4LWWG3RM4JTLZGNZ3XJDHEOZO7XW5R7BHQQJ6Z4Y47PZCEZHN6A, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.567664820 (376h45m37.338306156s) [noise=true , address=77.90.31.14:4692 , piece_id=NHO2352W7MMVGRS3MNPWSEFHWBOK73SQ6JQJEWZ5TV5UDFY44H5Q, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.568760968 (376h45m37.339402303s) [noise=true , address=77.90.31.10:4539 , piece_id=OI7V6HU5WMYZSQCS5PAN6OUCAJY4UGUZVTESMLB4LHOQJVMKJLHQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.573315612 (376h45m37.343956947s) [noise=true , address=77.90.31.15:4712 , piece_id=3DN3NPUYRN4AIBYBV2UCGE4TTYXD2WJRLHDMQEFIU36HBTYDOJ5Q, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.577761728 (376h45m37.348403053s) [noise=true , address=77.90.31.19:4850 , piece_id=DNYDFQHORBVHQTPKFLWEGQF6QRLE6IIE6YGTRP6SS63CPWVKRFFA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.579676210 (376h45m37.350317545s) [noise=true , address=77.90.31.12:4615 , piece_id=GLIW3756SYGXSX44VFMMDNYVEJPS6O4ZBNS53OJRV67QNGMGLPUA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.595175322 (376h45m37.365816657s) [noise=true , address=77.90.31.16:4743 , piece_id=2VSOXKLN6Q434ET7JYLRSU6BW7PIKNZ43MZOQ4HQL6MDP3WUJOZA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.601273909 (376h45m37.371915244s) [noise=true , address=77.90.31.13:4643 , piece_id=PO24YVCOVVNORI7NRD3CDUC4ZPRULAXUESGBEDHJZYYDJ2ZUPQDA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:07.782538378 (376h45m42.553179714s) [noise=true , address=45.91.116.5:4322 , piece_id=QMOE7Z74V4HHQO7LIN6KAJHEUSDKIAVC4XLFH3AOJMC2ILP6I4FA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:22.662088184 (376h45m57.432729519s) [noise=true , address=104.36.82.17:4333 , piece_id=TPALHRSYWC3XKW77CPA36EZOOJHPLUEN4V6DWFN5NHLIAFSN5KRQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:24.053592013 (376h45m58.824233348s) [noise=true , address=104.36.82.2:4757 , piece_id=JMJIECO7WCFHWASHRCTEAG6VCU4S25SEETORWGJJQ63VAW533YJA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:44.900784468 (376h46m19.671425793s) [noise=true , address=104.36.82.23:4473 , piece_id=V3RAX7LZBU744EQQXDEG6Z25UUGQUFSSQCWCXNZKNGCTHD375GCA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:47.726372911 (376h46m22.497014236s) [noise=true , address=104.36.82.60:4364 , piece_id=7WZJU4UIO4FLPUNWUH2PVIIH3ZWWVZCP2D2UUSCNBCIS727VVWPA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:56.047624046 (376h46m30.818265381s) [noise=true , address=104.36.82.3:4791 , piece_id=NG7KRWBROBZWNP63A4734KWSRNGFVGLLXFZ5PGXZDJXFDTZNCZLQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:56.047644345 (376h46m30.81828568s) [seg_pos={0 0}, upload=301417]: Segment reached optimal threshold of 14 pieces.
Jul 9 01:56:56.047718136 (376h46m30.818359471s) [noise=true , address=104.36.82.38:4708 , piece_id=RDIWOSET3Q2JKHRQ7HVUSVMNILHVEDQ7XYDL5DMWK37Q7YTRADQA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BdJtXzvjMoNoQUjtHETyADPcZBTzcFzH8YDd651t7eTiFb1wf): protocol: expected piece hash; context canceled; EOF
Jul 9 01:56:56.047756840 (376h46m30.818398175s) [noise=true , address=104.36.82.22:4532 , piece_id=3BRLUKXPLFBXIPKURV3BBSTA562UWWQBZ75HNYXEX75E74FKJIGQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cHUuyc42zpw5k9Lz5Qo9AW9DMvKFPxRNdKZDZtjFaeyzT4Nkp): protocol: expected piece hash; context canceled; EOF
Jul 9 01:56:56.047948587 (376h46m30.818589922s) [noise=true , address=104.36.82.21:4508 , piece_id=VVX5U234EZZNCZWIVYJVYBFDYG2SCKWS5HT5GM4H7CNNJQXDHT3Q, seg_pos={0 0}, upload=301417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1269NRnqpabU6H8q9vSCmE8yMzznD1L7dyZFmzCWhtzJaTjbXg3): protocol: write tcp 188.166.225.170:60172->104.36.82.21:4508: use of closed network connection; write tcp 188.166.225.170:60172->104.36.82.21:4508: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60172->104.36.82.21:4508: use of closed network connection
Jul 9 01:56:56.047987552 (376h46m30.818628887s) [noise=true , address=104.36.82.6:4880 , piece_id=NKXAOUHYMQ4WR6V6ZGQOKRZWLAGI44AF4SNUR4OKUNJNICD4UUNQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125NsQhq7kRmTjKS8nuc51PZddApFEi5C38FzC5pEX2QrWYKzSW): protocol: write tcp 188.166.225.170:38944->104.36.82.6:4880: use of closed network connection; write tcp 188.166.225.170:38944->104.36.82.6:4880: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38944->104.36.82.6:4880: use of closed network connection
Jul 9 01:56:56.047996138 (376h46m30.818637463s) [seg_pos={0 0}, upload=301417]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 01:56:56.048046284 (376h46m30.818687619s) [upload=301417]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 01:56:56.048050091 (376h46m30.818691416s) [upload=301417]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 01:56:56.339173192 (376h46m31.109814527s) [upload=301417]: Done uploading segments...
Jul 9 01:56:56.339190416 (376h46m31.109831751s) [upload=301417]: Upload finished. err: <nil>
Duration: 37.223271587s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
16 out of 60 is failed (threshold=3) [...........X......XX.XX..X......XXX....XX.....XX...X.XX.....] duration:
52.597729884s
FAIL (2025-07-09 02:00:29 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 52.597757036s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [.......................X......X..........X.............X....] duration:
35.868635076s
FAIL (2025-07-09 01:59:59 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
hRequestItem_SegmentBegin
Jul 9 01:56:00.629444016 (376h45m35.400085361s) [seg_pos={0 0}, upload=301417]: Begin upload segment...
Jul 9 01:56:00.629512196 (376h45m35.400153531s) [seg_pos={0 0}, upload=301417]: Done begin upload segment.
Jul 9 01:56:00.629572552 (376h45m35.400213887s) [noise=true , address=104.36.82.22:4532 , piece_id=3BRLUKXPLFBXIPKURV3BBSTA562UWWQBZ75HNYXEX75E74FKJIGQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.629579284 (376h45m35.400220619s) [noise=true , address=77.90.31.10:4539 , piece_id=OI7V6HU5WMYZSQCS5PAN6OUCAJY4UGUZVTESMLB4LHOQJVMKJLHQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.629690497 (376h45m35.400331832s) [noise=true , address=104.36.82.17:4333 , piece_id=TPALHRSYWC3XKW77CPA36EZOOJHPLUEN4V6DWFN5NHLIAFSN5KRQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630082818 (376h45m35.400724153s) [noise=true , address=104.36.82.6:4880 , piece_id=NKXAOUHYMQ4WR6V6ZGQOKRZWLAGI44AF4SNUR4OKUNJNICD4UUNQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630158242 (376h45m35.400799577s) [noise=true , address=79.127.201.214:4308 , piece_id=V4LWWG3RM4JTLZGNZ3XJDHEOZO7XW5R7BHQQJ6Z4Y47PZCEZHN6A, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630428680 (376h45m35.401070025s) [noise=true , address=77.90.31.12:4615 , piece_id=GLIW3756SYGXSX44VFMMDNYVEJPS6O4ZBNS53OJRV67QNGMGLPUA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630538129 (376h45m35.401179464s) [noise=true , address=104.36.82.60:4364 , piece_id=7WZJU4UIO4FLPUNWUH2PVIIH3ZWWVZCP2D2UUSCNBCIS727VVWPA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630862820 (376h45m35.401504165s) [noise=true , address=104.36.82.38:4708 , piece_id=RDIWOSET3Q2JKHRQ7HVUSVMNILHVEDQ7XYDL5DMWK37Q7YTRADQA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.631210084 (376h45m35.401851419s) [noise=true , address=104.36.82.21:4508 , piece_id=VVX5U234EZZNCZWIVYJVYBFDYG2SCKWS5HT5GM4H7CNNJQXDHT3Q, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.631555495 (376h45m35.40219684s) [noise=true , address=77.90.31.16:4743 , piece_id=2VSOXKLN6Q434ET7JYLRSU6BW7PIKNZ43MZOQ4HQL6MDP3WUJOZA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.631989165 (376h45m35.4026305s) [noise=true , address=104.36.82.23:4473 , piece_id=V3RAX7LZBU744EQQXDEG6Z25UUGQUFSSQCWCXNZKNGCTHD375GCA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.632258450 (376h45m35.402899785s) [noise=true , address=77.90.31.19:4850 , piece_id=DNYDFQHORBVHQTPKFLWEGQF6QRLE6IIE6YGTRP6SS63CPWVKRFFA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.632521984 (376h45m35.403163309s) [noise=true , address=45.91.116.5:4322 , piece_id=QMOE7Z74V4HHQO7LIN6KAJHEUSDKIAVC4XLFH3AOJMC2ILP6I4FA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.632835304 (376h45m35.403476649s) [noise=true , address=104.36.82.3:4791 , piece_id=NG7KRWBROBZWNP63A4734KWSRNGFVGLLXFZ5PGXZDJXFDTZNCZLQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.633120529 (376h45m35.403761864s) [noise=true , address=77.90.31.14:4692 , piece_id=NHO2352W7MMVGRS3MNPWSEFHWBOK73SQ6JQJEWZ5TV5UDFY44H5Q, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.636119667 (376h45m35.406761002s) [noise=true , address=104.36.82.2:4757 , piece_id=JMJIECO7WCFHWASHRCTEAG6VCU4S25SEETORWGJJQ63VAW533YJA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.636833744 (376h45m35.407475079s) [noise=true , address=77.90.31.15:4712 , piece_id=3DN3NPUYRN4AIBYBV2UCGE4TTYXD2WJRLHDMQEFIU36HBTYDOJ5Q, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.637112587 (376h45m35.407753922s) [noise=true , address=77.90.31.13:4643 , piece_id=PO24YVCOVVNORI7NRD3CDUC4ZPRULAXUESGBEDHJZYYDJ2ZUPQDA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:02.137197163 (376h45m36.907838498s) [upload=301417]: Next returned nil segment
Jul 9 01:56:02.137214166 (376h45m36.907855501s) [upload=301417]: Waiting for error group managing segments...
Jul 9 01:56:02.521317983 (376h45m37.291959318s) [noise=true , address=79.127.201.214:4308 , piece_id=V4LWWG3RM4JTLZGNZ3XJDHEOZO7XW5R7BHQQJ6Z4Y47PZCEZHN6A, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.567664820 (376h45m37.338306156s) [noise=true , address=77.90.31.14:4692 , piece_id=NHO2352W7MMVGRS3MNPWSEFHWBOK73SQ6JQJEWZ5TV5UDFY44H5Q, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.568760968 (376h45m37.339402303s) [noise=true , address=77.90.31.10:4539 , piece_id=OI7V6HU5WMYZSQCS5PAN6OUCAJY4UGUZVTESMLB4LHOQJVMKJLHQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.573315612 (376h45m37.343956947s) [noise=true , address=77.90.31.15:4712 , piece_id=3DN3NPUYRN4AIBYBV2UCGE4TTYXD2WJRLHDMQEFIU36HBTYDOJ5Q, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.577761728 (376h45m37.348403053s) [noise=true , address=77.90.31.19:4850 , piece_id=DNYDFQHORBVHQTPKFLWEGQF6QRLE6IIE6YGTRP6SS63CPWVKRFFA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.579676210 (376h45m37.350317545s) [noise=true , address=77.90.31.12:4615 , piece_id=GLIW3756SYGXSX44VFMMDNYVEJPS6O4ZBNS53OJRV67QNGMGLPUA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.595175322 (376h45m37.365816657s) [noise=true , address=77.90.31.16:4743 , piece_id=2VSOXKLN6Q434ET7JYLRSU6BW7PIKNZ43MZOQ4HQL6MDP3WUJOZA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.601273909 (376h45m37.371915244s) [noise=true , address=77.90.31.13:4643 , piece_id=PO24YVCOVVNORI7NRD3CDUC4ZPRULAXUESGBEDHJZYYDJ2ZUPQDA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:07.782538378 (376h45m42.553179714s) [noise=true , address=45.91.116.5:4322 , piece_id=QMOE7Z74V4HHQO7LIN6KAJHEUSDKIAVC4XLFH3AOJMC2ILP6I4FA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:22.662088184 (376h45m57.432729519s) [noise=true , address=104.36.82.17:4333 , piece_id=TPALHRSYWC3XKW77CPA36EZOOJHPLUEN4V6DWFN5NHLIAFSN5KRQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:24.053592013 (376h45m58.824233348s) [noise=true , address=104.36.82.2:4757 , piece_id=JMJIECO7WCFHWASHRCTEAG6VCU4S25SEETORWGJJQ63VAW533YJA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:44.900784468 (376h46m19.671425793s) [noise=true , address=104.36.82.23:4473 , piece_id=V3RAX7LZBU744EQQXDEG6Z25UUGQUFSSQCWCXNZKNGCTHD375GCA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:47.726372911 (376h46m22.497014236s) [noise=true , address=104.36.82.60:4364 , piece_id=7WZJU4UIO4FLPUNWUH2PVIIH3ZWWVZCP2D2UUSCNBCIS727VVWPA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:56.047624046 (376h46m30.818265381s) [noise=true , address=104.36.82.3:4791 , piece_id=NG7KRWBROBZWNP63A4734KWSRNGFVGLLXFZ5PGXZDJXFDTZNCZLQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:56.047644345 (376h46m30.81828568s) [seg_pos={0 0}, upload=301417]: Segment reached optimal threshold of 14 pieces.
Jul 9 01:56:56.047718136 (376h46m30.818359471s) [noise=true , address=104.36.82.38:4708 , piece_id=RDIWOSET3Q2JKHRQ7HVUSVMNILHVEDQ7XYDL5DMWK37Q7YTRADQA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BdJtXzvjMoNoQUjtHETyADPcZBTzcFzH8YDd651t7eTiFb1wf): protocol: expected piece hash; context canceled; EOF
Jul 9 01:56:56.047756840 (376h46m30.818398175s) [noise=true , address=104.36.82.22:4532 , piece_id=3BRLUKXPLFBXIPKURV3BBSTA562UWWQBZ75HNYXEX75E74FKJIGQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cHUuyc42zpw5k9Lz5Qo9AW9DMvKFPxRNdKZDZtjFaeyzT4Nkp): protocol: expected piece hash; context canceled; EOF
Jul 9 01:56:56.047948587 (376h46m30.818589922s) [noise=true , address=104.36.82.21:4508 , piece_id=VVX5U234EZZNCZWIVYJVYBFDYG2SCKWS5HT5GM4H7CNNJQXDHT3Q, seg_pos={0 0}, upload=301417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1269NRnqpabU6H8q9vSCmE8yMzznD1L7dyZFmzCWhtzJaTjbXg3): protocol: write tcp 188.166.225.170:60172->104.36.82.21:4508: use of closed network connection; write tcp 188.166.225.170:60172->104.36.82.21:4508: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60172->104.36.82.21:4508: use of closed network connection
Jul 9 01:56:56.047987552 (376h46m30.818628887s) [noise=true , address=104.36.82.6:4880 , piece_id=NKXAOUHYMQ4WR6V6ZGQOKRZWLAGI44AF4SNUR4OKUNJNICD4UUNQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125NsQhq7kRmTjKS8nuc51PZddApFEi5C38FzC5pEX2QrWYKzSW): protocol: write tcp 188.166.225.170:38944->104.36.82.6:4880: use of closed network connection; write tcp 188.166.225.170:38944->104.36.82.6:4880: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38944->104.36.82.6:4880: use of closed network connection
Jul 9 01:56:56.047996138 (376h46m30.818637463s) [seg_pos={0 0}, upload=301417]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 01:56:56.048046284 (376h46m30.818687619s) [upload=301417]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 01:56:56.048050091 (376h46m30.818691416s) [upload=301417]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 01:56:56.339173192 (376h46m31.109814527s) [upload=301417]: Done uploading segments...
Jul 9 01:56:56.339190416 (376h46m31.109831751s) [upload=301417]: Upload finished. err: <nil>
Duration: 35.868662547s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [....X...X.X..X...X..X............X......X..X...X.X.X.X...X..] duration:
10.733086044s
FAIL (2025-07-09 01:59:47 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 10.733172088s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.............X......................X.................X.....] duration:
7.138602285s
FAIL (2025-07-09 01:59:39 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
em: *pb.BatchRequestItem_SegmentCommit
Jul 9 01:55:26.378238935 (376h45m1.14888026s) [upload=301404]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 01:55:26.681901575 (376h45m1.45254291s) [upload=301404]: Done uploading segments...
Jul 9 01:55:26.681918448 (376h45m1.452559783s) [upload=301404]: Upload finished. err: <nil>
Jul 9 01:56:07.557068843 (376h45m42.327710178s) [upload=301419]: Starting upload
Jul 9 01:56:07.557123908 (376h45m42.327765243s) [upload=301419]: Done starting upload
Jul 9 01:56:07.557294845 (376h45m42.32793618s) [upload=301419]: Uploading segments...
Jul 9 01:56:07.557334942 (376h45m42.327976267s) [upload=301419]: Got next segment. Inline: false
Jul 9 01:56:07.557358848 (376h45m42.328000183s) [upload=301419]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 01:56:07.557363737 (376h45m42.328005072s) [upload=301419]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 01:56:07.786768822 (376h45m42.557410157s) [seg_pos={0 0}, upload=301419]: Begin upload segment...
Jul 9 01:56:07.786807697 (376h45m42.557449032s) [seg_pos={0 0}, upload=301419]: Done begin upload segment.
Jul 9 01:56:07.786827744 (376h45m42.557469069s) [noise=true , address=45.91.116.13:4154 , piece_id=WT7WL2P7RMJFA7XIZ3KNRVJ2RWDQGQBCM3J65X2X3MQ6WSAKHOIQ, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.786867240 (376h45m42.557508576s) [noise=true , address=104.36.82.2:4732 , piece_id=HF35AWQTHK4B5XQXLUOM5HPEBINSUIXGA3AAMZFEHZW4GDRDT4KQ, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.787161643 (376h45m42.557802968s) [noise=true , address=104.36.82.16:4294 , piece_id=TWDA4QZ5WAQF3OOXDDNAHBKYM5OAKBZ34QWW2FYXHJ6KGKDTDFRQ, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.787243289 (376h45m42.557884614s) [noise=true , address=77.90.31.11:4555 , piece_id=A7UGOPRF44445XR4OBEODVMMJU4HED7JUOPORHJKXVEPJO6KYIWQ, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.787405490 (376h45m42.558046815s) [noise=true , address=77.90.31.10:4518 , piece_id=ZC7HW4EDPE6UU2MLOHGHMYKBA4NXTJEL4I6CP6FK62QIAOEUT4ZQ, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.787501334 (376h45m42.558142669s) [noise=true , address=45.91.116.6:4975 , piece_id=MRETRUDTUOXXPHG4P25RMAJP4EDX4SS4B7O22PPPIQBSULZ52KNA, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.787663624 (376h45m42.558304959s) [noise=true , address=104.36.82.4:4801 , piece_id=OVVFQSK4TYRAXFNR6NZ74463CDC5BPKVROPU45ATGOTCZDUFDWHQ, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.787980670 (376h45m42.558622005s) [noise=true , address=45.91.116.11:4117 , piece_id=VDNXPWCQBXOOEAEJB7YB75AWP25A2AMCLTOT6KHFETFUXZ6UH3IQ, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.788245387 (376h45m42.558886712s) [noise=true , address=104.36.82.10:4078 , piece_id=ORPJSLOMY4IAB2NWYOQ7RCZCLEIQ25CIXFKKTWW5LBO5BFPPY64Q, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.788473754 (376h45m42.559115079s) [noise=true , address=45.91.116.7:4009 , piece_id=EEGGWRYW7EDJOVZDWO36KAGPFBQFSK5FJ7YU2FPBBWSIGV2KM4UA, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.788656052 (376h45m42.559297387s) [noise=true , address=104.36.82.6:4876 , piece_id=TUZF2S3DJSUK6MMNEQYWR35I56NAFLDRFPL646TIAGIUNVWE2ZGA, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.788848981 (376h45m42.559490316s) [noise=true , address=45.91.116.10:4081 , piece_id=4ZKRZFKWSILUJ3QNHIDRBGUETYMO7UZI5G5XR64XI2ANHXRCKAZQ, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.789032262 (376h45m42.559673597s) [noise=true , address=104.36.82.9:4006 , piece_id=P7YEMWPREVLDDFE7JDYAOEZJIYCAL3OAKKJJDC6EBMLLYSVRL73Q, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.789272181 (376h45m42.559913506s) [noise=true , address=77.90.31.19:4843 , piece_id=5FRTA2EJNZVNHV47EWYRFUH6VW2FJN2EBAKBRJWMP66BXF6SZF6A, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.789459429 (376h45m42.560100764s) [noise=true , address=45.91.116.8:4047 , piece_id=VISW3TARCTQMHB3662HBA2I7UW4IIREOJ32XOSBNTRPPTB4KD77A, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:07.789640225 (376h45m42.56028155s) [noise=true , address=77.90.31.21:4877 , piece_id=EN7QS3CEXXIBY7FU6Y32LLDKYCSVRJQ4G7RXRE2DV7CUPMZBVHDQ, seg_pos={0 0}, upload=301419]: Uploading piece...
Jul 9 01:56:09.422697440 (376h45m44.193338775s) [upload=301419]: Next returned nil segment
Jul 9 01:56:09.422714812 (376h45m44.193356148s) [upload=301419]: Waiting for error group managing segments...
Jul 9 01:56:09.703007945 (376h45m44.47364928s) [noise=true , address=104.36.82.9:4006 , piece_id=P7YEMWPREVLDDFE7JDYAOEZJIYCAL3OAKKJJDC6EBMLLYSVRL73Q, seg_pos={0 0}, upload=301419]: Done uploading piece. err: <nil>
Jul 9 01:56:09.780867601 (376h45m44.551508936s) [noise=true , address=104.36.82.16:4294 , piece_id=TWDA4QZ5WAQF3OOXDDNAHBKYM5OAKBZ34QWW2FYXHJ6KGKDTDFRQ, seg_pos={0 0}, upload=301419]: Done uploading piece. err: <nil>
Jul 9 01:56:09.840231843 (376h45m44.610873178s) [noise=true , address=45.91.116.6:4975 , piece_id=MRETRUDTUOXXPHG4P25RMAJP4EDX4SS4B7O22PPPIQBSULZ52KNA, seg_pos={0 0}, upload=301419]: Done uploading piece. err: <nil>
Jul 9 01:56:09.841984998 (376h45m44.612626363s) [noise=true , address=45.91.116.10:4081 , piece_id=4ZKRZFKWSILUJ3QNHIDRBGUETYMO7UZI5G5XR64XI2ANHXRCKAZQ, seg_pos={0 0}, upload=301419]: Done uploading piece. err: <nil>
Jul 9 01:56:09.850995516 (376h45m44.621636851s) [noise=true , address=45.91.116.13:4154 , piece_id=WT7WL2P7RMJFA7XIZ3KNRVJ2RWDQGQBCM3J65X2X3MQ6WSAKHOIQ, seg_pos={0 0}, upload=301419]: Done uploading piece. err: <nil>
Jul 9 01:56:09.860567239 (376h45m44.631208574s) [noise=true , address=45.91.116.7:4009 , piece_id=EEGGWRYW7EDJOVZDWO36KAGPFBQFSK5FJ7YU2FPBBWSIGV2KM4UA, seg_pos={0 0}, upload=301419]: Done uploading piece. err: <nil>
Jul 9 01:56:09.871129736 (376h45m44.641771071s) [noise=true , address=77.90.31.19:4843 , piece_id=5FRTA2EJNZVNHV47EWYRFUH6VW2FJN2EBAKBRJWMP66BXF6SZF6A, seg_pos={0 0}, upload=301419]: Done uploading piece. err: <nil>
Jul 9 01:56:09.874489343 (376h45m44.645130678s) [noise=true , address=77.90.31.11:4555 , piece_id=A7UGOPRF44445XR4OBEODVMMJU4HED7JUOPORHJKXVEPJO6KYIWQ, seg_pos={0 0}, upload=301419]: Done uploading piece. err: <nil>
Jul 9 01:56:09.875282992 (376h45m44.645924327s) [noise=true , address=77.90.31.21:4877 , piece_id=EN7QS3CEXXIBY7FU6Y32LLDKYCSVRJQ4G7RXRE2DV7CUPMZBVHDQ, seg_pos={0 0}, upload=301419]: Done uploading piece. err: <nil>
Jul 9 01:56:09.882277543 (376h45m44.652918878s) [noise=true , address=77.90.31.10:4518 , piece_id=ZC7HW4EDPE6UU2MLOHGHMYKBA4NXTJEL4I6CP6FK62QIAOEUT4ZQ, seg_pos={0 0}, upload=301419]: Done uploading piece. err: <nil>
Jul 9 01:56:09.909865152 (376h45m44.680506487s) [noise=true , address=45.91.116.11:4117 , piece_id=VDNXPWCQBXOOEAEJB7YB75AWP25A2AMCLTOT6KHFETFUXZ6UH3IQ, seg_pos={0 0}, upload=301419]: Done uploading piece. err: <nil>
Jul 9 01:56:09.930049268 (376h45m44.700690603s) [noise=true , address=45.91.116.8:4047 , piece_id=VISW3TARCTQMHB3662HBA2I7UW4IIREOJ32XOSBNTRPPTB4KD77A, seg_pos={0 0}, upload=301419]: Done uploading piece. err: <nil>
Jul 9 01:56:36.916221275 (376h46m11.68686263s) [noise=true , address=104.36.82.6:4876 , piece_id=TUZF2S3DJSUK6MMNEQYWR35I56NAFLDRFPL646TIAGIUNVWE2ZGA, seg_pos={0 0}, upload=301419]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 188.166.225.170:36068->104.36.82.6:4876: use of closed network connection; write tcp 188.166.225.170:36068->104.36.82.6:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36068->104.36.82.6:4876: use of closed network connection
Jul 9 01:56:36.916318731 (376h46m11.686960076s) [noise=true , address=104.36.82.4:4801 , piece_id=OVVFQSK4TYRAXFNR6NZ74463CDC5BPKVROPU45ATGOTCZDUFDWHQ, seg_pos={0 0}, upload=301419]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 188.166.225.170:36588->104.36.82.4:4801: use of closed network connection; context deadline exceeded
Jul 9 01:56:36.916361463 (376h46m11.687002818s) [noise=true , address=104.36.82.2:4732 , piece_id=HF35AWQTHK4B5XQXLUOM5HPEBINSUIXGA3AAMZFEHZW4GDRDT4KQ, seg_pos={0 0}, upload=301419]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: write tcp 188.166.225.170:44126->104.36.82.2:4732: use of closed network connection; write tcp 188.166.225.170:44126->104.36.82.2:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44126->104.36.82.2:4732: use of closed network connection
Jul 9 01:56:36.916428551 (376h46m11.687069916s) [noise=true , address=104.36.82.10:4078 , piece_id=ORPJSLOMY4IAB2NWYOQ7RCZCLEIQ25CIXFKKTWW5LBO5BFPPY64Q, seg_pos={0 0}, upload=301419]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 188.166.225.170:53862->104.36.82.10:4078: use of closed network connection; write tcp 188.166.225.170:53862->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53862->104.36.82.10:4078: use of closed network connection
Jul 9 01:56:36.916470932 (376h46m11.687112338s) [seg_pos={0 0}, upload=301419]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 01:56:37.151140589 (376h46m11.921781964s) [upload=301419]: Done uploading segments...
Jul 9 01:56:37.151169034 (376h46m11.921810369s) [upload=301419]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 7.138633103s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [........................X......X..........X.............X...] duration:
46.76130419s
FAIL (2025-07-09 01:59:11 UTC)
orbiter warning:
======ERROR======
uplink: context deadline exceeded
======LOGS======
hRequestItem_SegmentBegin
Jul 9 01:56:00.629444016 (376h45m35.400085361s) [seg_pos={0 0}, upload=301417]: Begin upload segment...
Jul 9 01:56:00.629512196 (376h45m35.400153531s) [seg_pos={0 0}, upload=301417]: Done begin upload segment.
Jul 9 01:56:00.629572552 (376h45m35.400213887s) [noise=true , address=104.36.82.22:4532 , piece_id=3BRLUKXPLFBXIPKURV3BBSTA562UWWQBZ75HNYXEX75E74FKJIGQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.629579284 (376h45m35.400220619s) [noise=true , address=77.90.31.10:4539 , piece_id=OI7V6HU5WMYZSQCS5PAN6OUCAJY4UGUZVTESMLB4LHOQJVMKJLHQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.629690497 (376h45m35.400331832s) [noise=true , address=104.36.82.17:4333 , piece_id=TPALHRSYWC3XKW77CPA36EZOOJHPLUEN4V6DWFN5NHLIAFSN5KRQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630082818 (376h45m35.400724153s) [noise=true , address=104.36.82.6:4880 , piece_id=NKXAOUHYMQ4WR6V6ZGQOKRZWLAGI44AF4SNUR4OKUNJNICD4UUNQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630158242 (376h45m35.400799577s) [noise=true , address=79.127.201.214:4308 , piece_id=V4LWWG3RM4JTLZGNZ3XJDHEOZO7XW5R7BHQQJ6Z4Y47PZCEZHN6A, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630428680 (376h45m35.401070025s) [noise=true , address=77.90.31.12:4615 , piece_id=GLIW3756SYGXSX44VFMMDNYVEJPS6O4ZBNS53OJRV67QNGMGLPUA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630538129 (376h45m35.401179464s) [noise=true , address=104.36.82.60:4364 , piece_id=7WZJU4UIO4FLPUNWUH2PVIIH3ZWWVZCP2D2UUSCNBCIS727VVWPA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.630862820 (376h45m35.401504165s) [noise=true , address=104.36.82.38:4708 , piece_id=RDIWOSET3Q2JKHRQ7HVUSVMNILHVEDQ7XYDL5DMWK37Q7YTRADQA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.631210084 (376h45m35.401851419s) [noise=true , address=104.36.82.21:4508 , piece_id=VVX5U234EZZNCZWIVYJVYBFDYG2SCKWS5HT5GM4H7CNNJQXDHT3Q, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.631555495 (376h45m35.40219684s) [noise=true , address=77.90.31.16:4743 , piece_id=2VSOXKLN6Q434ET7JYLRSU6BW7PIKNZ43MZOQ4HQL6MDP3WUJOZA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.631989165 (376h45m35.4026305s) [noise=true , address=104.36.82.23:4473 , piece_id=V3RAX7LZBU744EQQXDEG6Z25UUGQUFSSQCWCXNZKNGCTHD375GCA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.632258450 (376h45m35.402899785s) [noise=true , address=77.90.31.19:4850 , piece_id=DNYDFQHORBVHQTPKFLWEGQF6QRLE6IIE6YGTRP6SS63CPWVKRFFA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.632521984 (376h45m35.403163309s) [noise=true , address=45.91.116.5:4322 , piece_id=QMOE7Z74V4HHQO7LIN6KAJHEUSDKIAVC4XLFH3AOJMC2ILP6I4FA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.632835304 (376h45m35.403476649s) [noise=true , address=104.36.82.3:4791 , piece_id=NG7KRWBROBZWNP63A4734KWSRNGFVGLLXFZ5PGXZDJXFDTZNCZLQ, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.633120529 (376h45m35.403761864s) [noise=true , address=77.90.31.14:4692 , piece_id=NHO2352W7MMVGRS3MNPWSEFHWBOK73SQ6JQJEWZ5TV5UDFY44H5Q, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.636119667 (376h45m35.406761002s) [noise=true , address=104.36.82.2:4757 , piece_id=JMJIECO7WCFHWASHRCTEAG6VCU4S25SEETORWGJJQ63VAW533YJA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.636833744 (376h45m35.407475079s) [noise=true , address=77.90.31.15:4712 , piece_id=3DN3NPUYRN4AIBYBV2UCGE4TTYXD2WJRLHDMQEFIU36HBTYDOJ5Q, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:00.637112587 (376h45m35.407753922s) [noise=true , address=77.90.31.13:4643 , piece_id=PO24YVCOVVNORI7NRD3CDUC4ZPRULAXUESGBEDHJZYYDJ2ZUPQDA, seg_pos={0 0}, upload=301417]: Uploading piece...
Jul 9 01:56:02.137197163 (376h45m36.907838498s) [upload=301417]: Next returned nil segment
Jul 9 01:56:02.137214166 (376h45m36.907855501s) [upload=301417]: Waiting for error group managing segments...
Jul 9 01:56:02.521317983 (376h45m37.291959318s) [noise=true , address=79.127.201.214:4308 , piece_id=V4LWWG3RM4JTLZGNZ3XJDHEOZO7XW5R7BHQQJ6Z4Y47PZCEZHN6A, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.567664820 (376h45m37.338306156s) [noise=true , address=77.90.31.14:4692 , piece_id=NHO2352W7MMVGRS3MNPWSEFHWBOK73SQ6JQJEWZ5TV5UDFY44H5Q, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.568760968 (376h45m37.339402303s) [noise=true , address=77.90.31.10:4539 , piece_id=OI7V6HU5WMYZSQCS5PAN6OUCAJY4UGUZVTESMLB4LHOQJVMKJLHQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.573315612 (376h45m37.343956947s) [noise=true , address=77.90.31.15:4712 , piece_id=3DN3NPUYRN4AIBYBV2UCGE4TTYXD2WJRLHDMQEFIU36HBTYDOJ5Q, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.577761728 (376h45m37.348403053s) [noise=true , address=77.90.31.19:4850 , piece_id=DNYDFQHORBVHQTPKFLWEGQF6QRLE6IIE6YGTRP6SS63CPWVKRFFA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.579676210 (376h45m37.350317545s) [noise=true , address=77.90.31.12:4615 , piece_id=GLIW3756SYGXSX44VFMMDNYVEJPS6O4ZBNS53OJRV67QNGMGLPUA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.595175322 (376h45m37.365816657s) [noise=true , address=77.90.31.16:4743 , piece_id=2VSOXKLN6Q434ET7JYLRSU6BW7PIKNZ43MZOQ4HQL6MDP3WUJOZA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:02.601273909 (376h45m37.371915244s) [noise=true , address=77.90.31.13:4643 , piece_id=PO24YVCOVVNORI7NRD3CDUC4ZPRULAXUESGBEDHJZYYDJ2ZUPQDA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:07.782538378 (376h45m42.553179714s) [noise=true , address=45.91.116.5:4322 , piece_id=QMOE7Z74V4HHQO7LIN6KAJHEUSDKIAVC4XLFH3AOJMC2ILP6I4FA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:22.662088184 (376h45m57.432729519s) [noise=true , address=104.36.82.17:4333 , piece_id=TPALHRSYWC3XKW77CPA36EZOOJHPLUEN4V6DWFN5NHLIAFSN5KRQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:24.053592013 (376h45m58.824233348s) [noise=true , address=104.36.82.2:4757 , piece_id=JMJIECO7WCFHWASHRCTEAG6VCU4S25SEETORWGJJQ63VAW533YJA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:44.900784468 (376h46m19.671425793s) [noise=true , address=104.36.82.23:4473 , piece_id=V3RAX7LZBU744EQQXDEG6Z25UUGQUFSSQCWCXNZKNGCTHD375GCA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:47.726372911 (376h46m22.497014236s) [noise=true , address=104.36.82.60:4364 , piece_id=7WZJU4UIO4FLPUNWUH2PVIIH3ZWWVZCP2D2UUSCNBCIS727VVWPA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:56.047624046 (376h46m30.818265381s) [noise=true , address=104.36.82.3:4791 , piece_id=NG7KRWBROBZWNP63A4734KWSRNGFVGLLXFZ5PGXZDJXFDTZNCZLQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: <nil>
Jul 9 01:56:56.047644345 (376h46m30.81828568s) [seg_pos={0 0}, upload=301417]: Segment reached optimal threshold of 14 pieces.
Jul 9 01:56:56.047718136 (376h46m30.818359471s) [noise=true , address=104.36.82.38:4708 , piece_id=RDIWOSET3Q2JKHRQ7HVUSVMNILHVEDQ7XYDL5DMWK37Q7YTRADQA, seg_pos={0 0}, upload=301417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BdJtXzvjMoNoQUjtHETyADPcZBTzcFzH8YDd651t7eTiFb1wf): protocol: expected piece hash; context canceled; EOF
Jul 9 01:56:56.047756840 (376h46m30.818398175s) [noise=true , address=104.36.82.22:4532 , piece_id=3BRLUKXPLFBXIPKURV3BBSTA562UWWQBZ75HNYXEX75E74FKJIGQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cHUuyc42zpw5k9Lz5Qo9AW9DMvKFPxRNdKZDZtjFaeyzT4Nkp): protocol: expected piece hash; context canceled; EOF
Jul 9 01:56:56.047948587 (376h46m30.818589922s) [noise=true , address=104.36.82.21:4508 , piece_id=VVX5U234EZZNCZWIVYJVYBFDYG2SCKWS5HT5GM4H7CNNJQXDHT3Q, seg_pos={0 0}, upload=301417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1269NRnqpabU6H8q9vSCmE8yMzznD1L7dyZFmzCWhtzJaTjbXg3): protocol: write tcp 188.166.225.170:60172->104.36.82.21:4508: use of closed network connection; write tcp 188.166.225.170:60172->104.36.82.21:4508: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60172->104.36.82.21:4508: use of closed network connection
Jul 9 01:56:56.047987552 (376h46m30.818628887s) [noise=true , address=104.36.82.6:4880 , piece_id=NKXAOUHYMQ4WR6V6ZGQOKRZWLAGI44AF4SNUR4OKUNJNICD4UUNQ, seg_pos={0 0}, upload=301417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125NsQhq7kRmTjKS8nuc51PZddApFEi5C38FzC5pEX2QrWYKzSW): protocol: write tcp 188.166.225.170:38944->104.36.82.6:4880: use of closed network connection; write tcp 188.166.225.170:38944->104.36.82.6:4880: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38944->104.36.82.6:4880: use of closed network connection
Jul 9 01:56:56.047996138 (376h46m30.818637463s) [seg_pos={0 0}, upload=301417]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 01:56:56.048046284 (376h46m30.818687619s) [upload=301417]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 01:56:56.048050091 (376h46m30.818691416s) [upload=301417]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 01:56:56.339173192 (376h46m31.109814527s) [upload=301417]: Done uploading segments...
Jul 9 01:56:56.339190416 (376h46m31.109831751s) [upload=301417]: Upload finished. err: <nil>
Duration: 46.761329497s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
16 out of 60 is failed (threshold=3) [............X......XX.XX..X......XXX....XX.....XX...X.XX....] duration:
44.617472596s
FAIL (2025-07-09 01:59:09 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 44.617492674s
core
uplink
us-select-1
us1-placement-20
|