Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [X.XX.X.X..XXXXX....X...X..X..X...X...X.X.........X.....X....] duration:
5.922091391s trace:
https://jaeger.dev.storj.io/trace/400f8ae82c545e27
FAIL (2025-12-28 14:57:49 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: P3XVLM2MNM224KOHB65XZBQNGQNOVJ37RY2FR4JSPRZPOFZM5W4Q) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: CG7G54I3IGFS32ZAY2WJHDRCYAGLFVO6TOVKX7R4CUJ6UO7V3G4Q) context deadline exceeded
======LOGS======
28 14:54:28.651637995 (551h25m28.52272104s) [upload=483478]: Starting upload
Dec 28 14:54:28.651775788 (551h25m28.522858823s) [upload=483478]: Done starting upload
Dec 28 14:54:28.651857663 (551h25m28.522940698s) [upload=483478]: Uploading segments...
Dec 28 14:54:28.651966430 (551h25m28.523049475s) [upload=483478]: Got next segment. Inline: false
Dec 28 14:54:28.652001437 (551h25m28.523084472s) [upload=483478]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 14:54:28.652006597 (551h25m28.523089632s) [upload=483478]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 14:54:28.897465992 (551h25m28.768549038s) [seg_pos={0 0}, upload=483478]: Begin upload segment...
Dec 28 14:54:28.897524775 (551h25m28.76860782s) [seg_pos={0 0}, upload=483478]: Done begin upload segment.
Dec 28 14:54:28.897584187 (551h25m28.768667232s) [noise=true , address=145.79.227.50:4373 , piece_id=F5HNDNMLFF4GW3EVG2DR2V2I7D6OZN7NA7B6WZMKWGTSZ5FPKREA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.897595700 (551h25m28.768678735s) [noise=true , address=45.91.116.7:4010 , piece_id=P3XVLM2MNM224KOHB65XZBQNGQNOVJ37RY2FR4JSPRZPOFZM5W4Q, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.898192205 (551h25m28.76927524s) [noise=true , address=45.91.116.6:4973 , piece_id=ZCNSUED6MUMN4OCGGVXD3RVV6OI3D3DRDRLD43YGE2WVDDTGW3FQ, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.898681176 (551h25m28.769764212s) [noise=true , address=145.79.227.18:4007 , piece_id=Y52XM3XDSZFKHR6C7CYQ7VIUQGMWNB3JQMGPRQUVKENFJ7RRSQYQ, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.899027356 (551h25m28.770110391s) [noise=true , address=145.79.227.21:4473 , piece_id=44NH6BNBDRVJMYAJDGHIIWRPHQZGHCA3SIUGQ4IHTAYSMW3U2ESA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.899416557 (551h25m28.770499602s) [noise=true , address=145.79.227.5:4223 , piece_id=T6JVT2OXKEBVYP4ICGLCPK3CLN5H25TFTRXC22E7OBD7NLA3HDLA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.899745403 (551h25m28.770828448s) [noise=true , address=45.91.116.5:4298 , piece_id=F6UV3BDNWFAGWBBJLNSBQETPARFEGKB4Z6POFKVBV6X5DNIFOODA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.900154752 (551h25m28.771237797s) [noise=true , address=145.79.227.44:4194 , piece_id=77E4L3WGPLKI2HUTR6UAOHYTVUMBBGMGVALT6R42L7PLMVIT7JJQ, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.900476626 (551h25m28.771559661s) [noise=true , address=145.79.227.15:4874 , piece_id=3CYMIHKUTOXHXJBVJKUUJC6HKTQA4AD5HCET3UJNDMFHGXGXIOOA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.902303083 (551h25m28.773386118s) [noise=true , address=145.79.227.36:4519 , piece_id=I7VEEU6WTZG4YRMJIKUQTWLPFBECYVYTV4Z43NP3U2QGHPGB3EEA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.903509890 (551h25m28.774592926s) [noise=true , address=77.90.31.13:4626 , piece_id=CG7G54I3IGFS32ZAY2WJHDRCYAGLFVO6TOVKX7R4CUJ6UO7V3G4Q, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.905006530 (551h25m28.776089565s) [noise=true , address=77.90.31.16:4735 , piece_id=Z2JJ7IIWX3RQLXM3HABIULPLEQSMDKVXC6F7YJCW3HCVEBRRYM6Q, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.906620755 (551h25m28.77770379s) [noise=true , address=145.79.227.52:4445 , piece_id=ZKUYH45CZIQNCWP5Q3JMEJTOA5TJSTZL7TTWOBGF5QOP245ZNHHQ, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.908401716 (551h25m28.779484761s) [noise=true , address=145.79.227.2:4113 , piece_id=PIBB5A34I4C3T5RX4DPHMSFBGJGZDGWOP7XBELPTOYA45DRVZZCA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.909977536 (551h25m28.781060571s) [noise=true , address=145.79.227.32:4835 , piece_id=FVGQX4522G64FUDBCKN6LXSTECVOXZONMO3WPUQGWYUMKKLHAL7Q, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.912006570 (551h25m28.783089625s) [noise=true , address=145.79.227.10:4401 , piece_id=A4UVXDWW66OY2DIFDGN47XYGK255NTLGR2U36NUDF32AOJD2LFKA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:30.616536389 (551h25m30.487619424s) [upload=483478]: Next returned nil segment
Dec 28 14:54:30.616555134 (551h25m30.487638179s) [upload=483478]: Waiting for error group managing segments...
Dec 28 14:54:30.902690564 (551h25m30.773773619s) [noise=true , address=145.79.227.5:4223 , piece_id=T6JVT2OXKEBVYP4ICGLCPK3CLN5H25TFTRXC22E7OBD7NLA3HDLA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.908186579 (551h25m30.779269624s) [noise=true , address=145.79.227.2:4113 , piece_id=PIBB5A34I4C3T5RX4DPHMSFBGJGZDGWOP7XBELPTOYA45DRVZZCA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.908476522 (551h25m30.779559557s) [noise=true , address=145.79.227.21:4473 , piece_id=44NH6BNBDRVJMYAJDGHIIWRPHQZGHCA3SIUGQ4IHTAYSMW3U2ESA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.910250058 (551h25m30.781333113s) [noise=true , address=145.79.227.18:4007 , piece_id=Y52XM3XDSZFKHR6C7CYQ7VIUQGMWNB3JQMGPRQUVKENFJ7RRSQYQ, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.911106399 (551h25m30.782189444s) [noise=true , address=145.79.227.50:4373 , piece_id=F5HNDNMLFF4GW3EVG2DR2V2I7D6OZN7NA7B6WZMKWGTSZ5FPKREA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.914846541 (551h25m30.785929587s) [noise=true , address=145.79.227.10:4401 , piece_id=A4UVXDWW66OY2DIFDGN47XYGK255NTLGR2U36NUDF32AOJD2LFKA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.919475566 (551h25m30.790558621s) [noise=true , address=145.79.227.52:4445 , piece_id=ZKUYH45CZIQNCWP5Q3JMEJTOA5TJSTZL7TTWOBGF5QOP245ZNHHQ, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.921006410 (551h25m30.792089455s) [noise=true , address=145.79.227.36:4519 , piece_id=I7VEEU6WTZG4YRMJIKUQTWLPFBECYVYTV4Z43NP3U2QGHPGB3EEA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.921040545 (551h25m30.7921236s) [noise=true , address=145.79.227.15:4874 , piece_id=3CYMIHKUTOXHXJBVJKUUJC6HKTQA4AD5HCET3UJNDMFHGXGXIOOA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.941624211 (551h25m30.812707256s) [noise=true , address=145.79.227.32:4835 , piece_id=FVGQX4522G64FUDBCKN6LXSTECVOXZONMO3WPUQGWYUMKKLHAL7Q, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:31.182363167 (551h25m31.053446202s) [noise=true , address=77.90.31.16:4735 , piece_id=Z2JJ7IIWX3RQLXM3HABIULPLEQSMDKVXC6F7YJCW3HCVEBRRYM6Q, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:31.261978689 (551h25m31.133061734s) [noise=true , address=77.90.31.13:4626 , piece_id=CG7G54I3IGFS32ZAY2WJHDRCYAGLFVO6TOVKX7R4CUJ6UO7V3G4Q, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:31.276375082 (551h25m31.147458117s) [noise=true , address=45.91.116.7:4010 , piece_id=P3XVLM2MNM224KOHB65XZBQNGQNOVJ37RY2FR4JSPRZPOFZM5W4Q, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:31.276394178 (551h25m31.147477223s) [seg_pos={0 0}, upload=483478]: Segment reached optimal threshold of 13 pieces.
Dec 28 14:54:31.276572508 (551h25m31.147655543s) [noise=true , address=45.91.116.6:4973 , piece_id=ZCNSUED6MUMN4OCGGVXD3RVV6OI3D3DRDRLD43YGE2WVDDTGW3FQ, seg_pos={0 0}, upload=483478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF
Dec 28 14:54:31.276578930 (551h25m31.147661975s) [noise=true , address=45.91.116.6:4973 , piece_id=ZCNSUED6MUMN4OCGGVXD3RVV6OI3D3DRDRLD43YGE2WVDDTGW3FQ, seg_pos={0 0}, upload=483478]: Long tail cancellation due to optimal threshold
Dec 28 14:54:31.276703968 (551h25m31.147787003s) [noise=true , address=45.91.116.5:4298 , piece_id=F6UV3BDNWFAGWBBJLNSBQETPARFEGKB4Z6POFKVBV6X5DNIFOODA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:34120->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:34120->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34120->45.91.116.5:4298: use of closed network connection
Dec 28 14:54:31.276709198 (551h25m31.147792233s) [noise=true , address=45.91.116.5:4298 , piece_id=F6UV3BDNWFAGWBBJLNSBQETPARFEGKB4Z6POFKVBV6X5DNIFOODA, seg_pos={0 0}, upload=483478]: Long tail cancellation due to optimal threshold
Dec 28 14:54:31.277040298 (551h25m31.148123343s) [noise=true , address=145.79.227.44:4194 , piece_id=77E4L3WGPLKI2HUTR6UAOHYTVUMBBGMGVALT6R42L7PLMVIT7JJQ, seg_pos={0 0}, upload=483478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF
Dec 28 14:54:31.277060938 (551h25m31.148143983s) [noise=true , address=145.79.227.44:4194 , piece_id=77E4L3WGPLKI2HUTR6UAOHYTVUMBBGMGVALT6R42L7PLMVIT7JJQ, seg_pos={0 0}, upload=483478]: Long tail cancellation due to optimal threshold
Dec 28 14:54:31.277105372 (551h25m31.148188417s) [seg_pos={0 0}, upload=483478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 14:54:31.277258323 (551h25m31.148341378s) [upload=483478]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 14:54:31.277265838 (551h25m31.148348903s) [upload=483478]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 14:54:31.575778626 (551h25m31.446861691s) [upload=483478]: Done uploading segments...
Dec 28 14:54:31.575803704 (551h25m31.446886749s) [upload=483478]: Upload finished. err: <nil>
Duration: 5.922120236s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [...................X...........X............................] duration:
15.807646155s trace:
https://jaeger.dev.storj.io/trace/70ba9ccd98ccb34e
FAIL (2025-12-28 14:57:46 UTC)
orbiter warning: context deadline exceeded
Duration: 15.807805288s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [....................X...........X...........................] duration:
15.054127699s trace:
https://jaeger.dev.storj.io/trace/70ba9ccd98ccb34e
FAIL (2025-12-28 14:57:25 UTC)
orbiter warning: context deadline exceeded
Duration: 15.054150692s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
19.931503425s trace:
https://jaeger.dev.storj.io/trace/35a5d4335e2cde85
FAIL (2025-12-28 14:57:19 UTC)
orbiter warning: context deadline exceeded
Duration: 19.931541768s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
47 out of 60 is failed (threshold=6) [.XXXXXXXXXXXXXXXXXXXXXXX.X.XX.XXXXXXXX.X..XXXXX.X.X.XXXXX...] duration:
5.000661272s trace:
https://jaeger.dev.storj.io/trace/131df032dcbcf2b
FAIL (2025-12-28 14:57:18 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000688684s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [...X...X............X...............X......X....X...........] duration:
54.623188679s trace:
https://jaeger.dev.storj.io/trace/1440a926e26dd799
FAIL (2025-12-28 14:56:53 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
noise=true , address=145.79.227.44:4212 , piece_id=V73IIIU7CCTULBPEXD2YWSFUUGEP2ETVL7LQGIWWEXICZ5SK4BQA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.239195729 (551h16m25.110278774s) [noise=true , address=145.79.227.34:4931 , piece_id=IBNDRK5G5MGWQKRTJ3CS7LI7F3TFUJDHZDG6HJZ44XOQAEWBVEBA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.239549302 (551h16m25.110632347s) [noise=true , address=64.49.13.140:4515 , piece_id=42YNEU3HOVPRPUVOCPKYSQUBTDLBQELQCWNIYJBDXD2RV2NRDJDQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.239966557 (551h16m25.111049602s) [noise=true , address=77.90.31.18:4816 , piece_id=J3DJPEVEESV6EA5KJU6TN7MO4ATQZDBQOSJSH5S5KSZRYGHHZRSQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.240145648 (551h16m25.111228683s) [noise=true , address=77.90.31.14:4688 , piece_id=LNETNIH6PNQ2OR2NQKBWWWOJEFYXIGYHHH34WW7DHTKKYQWJ6DLA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.240933478 (551h16m25.112016523s) [noise=true , address=77.90.31.19:4867 , piece_id=NWQD7PXUGDHUKRF4J3E6EUORCKUM3TDAD2EJH7F3CE55G4EA7SJQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.241389286 (551h16m25.112472331s) [noise=true , address=145.79.227.40:4075 , piece_id=XKJZ2GSNELWKMTMZA5EZ2YTM62XREZQDFI7MDDKAPE3YXRY4KWNA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.241451415 (551h16m25.11253445s) [noise=true , address=77.90.31.21:4889 , piece_id=WSIKD7GCMKHWEFYC7UZ3YWGQHQDIZVW2YJY4BD5ENMCYBJZJMXEQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.241843531 (551h16m25.112926566s) [noise=true , address=145.79.227.46:4608 , piece_id=T5G4T4OU62UPXPIACXJQEZ36O2N63HL4L4HRMW3EN6B4PIE3M7YA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.241911240 (551h16m25.112994285s) [noise=true , address=64.49.13.169:4711 , piece_id=YHY2V4EHCMQEDR6OTZE2POM45Y7FYVNOSEGSUOM5IYWCDWDLRIKQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.242196533 (551h16m25.113279578s) [noise=true , address=193.149.164.143:4697 , piece_id=6SHPKJO6AZLKYPIQWJWNI4YSAMLAVU7BGDGQLQC4I6AGCRWT52DA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.242532343 (551h16m25.113615378s) [noise=true , address=145.79.227.29:4947 , piece_id=FSJZQVSKN76O5RLIVQDPKAN2PDV4VLRCMBRA5EYJGFOETOZ6D7AQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.242582478 (551h16m25.113665613s) [noise=true , address=145.79.227.18:4010 , piece_id=GVY67LYRK3UZRX6OLLWC5JCJEGV3RCIS6DATIDMQCBJSL7DCAE4Q, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.242986888 (551h16m25.114069923s) [noise=true , address=64.49.13.27:4966 , piece_id=N3R7JO5PL47V7ZUXNLXFOHSI2MEJ2SVQNHE6M2DWDYF3D4L34ZCA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.243048475 (551h16m25.11413151s) [noise=true , address=77.90.31.11:4585 , piece_id=5MKOQOD3O7TXKXODJWTVZI5EAKBEO7JE3KKXD63F6SO3EBBSVD6A, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.243336464 (551h16m25.114419499s) [noise=true , address=77.90.31.17:4801 , piece_id=OWQPIGFIL5SFLUI77IYA3LKVUFPSBL6JSJO6XWZLQFV7GXHF3QOA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:26.537548687 (551h16m26.408631722s) [upload=483348]: Next returned nil segment
Dec 28 14:45:26.537559538 (551h16m26.408642573s) [upload=483348]: Waiting for error group managing segments...
Dec 28 14:45:26.899924119 (551h16m26.771007164s) [noise=true , address=64.49.13.140:4515 , piece_id=42YNEU3HOVPRPUVOCPKYSQUBTDLBQELQCWNIYJBDXD2RV2NRDJDQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:26.977957828 (551h16m26.849040873s) [noise=true , address=64.49.13.27:4966 , piece_id=N3R7JO5PL47V7ZUXNLXFOHSI2MEJ2SVQNHE6M2DWDYF3D4L34ZCA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.120995816 (551h16m26.992078862s) [noise=true , address=145.79.227.46:4608 , piece_id=T5G4T4OU62UPXPIACXJQEZ36O2N63HL4L4HRMW3EN6B4PIE3M7YA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.151904071 (551h16m27.022987116s) [noise=true , address=145.79.227.34:4931 , piece_id=IBNDRK5G5MGWQKRTJ3CS7LI7F3TFUJDHZDG6HJZ44XOQAEWBVEBA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.165435146 (551h16m27.036518181s) [noise=true , address=145.79.227.29:4947 , piece_id=FSJZQVSKN76O5RLIVQDPKAN2PDV4VLRCMBRA5EYJGFOETOZ6D7AQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.175130708 (551h16m27.046213754s) [noise=true , address=145.79.227.18:4010 , piece_id=GVY67LYRK3UZRX6OLLWC5JCJEGV3RCIS6DATIDMQCBJSL7DCAE4Q, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.352651587 (551h16m27.223734632s) [noise=true , address=77.90.31.19:4867 , piece_id=NWQD7PXUGDHUKRF4J3E6EUORCKUM3TDAD2EJH7F3CE55G4EA7SJQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.375485928 (551h16m27.246568973s) [noise=true , address=77.90.31.11:4585 , piece_id=5MKOQOD3O7TXKXODJWTVZI5EAKBEO7JE3KKXD63F6SO3EBBSVD6A, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.428947778 (551h16m27.300030823s) [noise=true , address=77.90.31.18:4816 , piece_id=J3DJPEVEESV6EA5KJU6TN7MO4ATQZDBQOSJSH5S5KSZRYGHHZRSQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.567872934 (551h16m27.438955969s) [noise=true , address=77.90.31.17:4801 , piece_id=OWQPIGFIL5SFLUI77IYA3LKVUFPSBL6JSJO6XWZLQFV7GXHF3QOA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.983670843 (551h16m27.854753878s) [noise=true , address=77.90.31.13:4650 , piece_id=NHTWFNKCRHELW7BJ2NR4GGHY4LU743RVLCCR236AO4MQUZD7KMEQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:28.019933813 (551h16m27.891016848s) [noise=true , address=145.79.227.30:4794 , piece_id=OZXLKN4V62NRD3WCMF65C7U7OPM2FPBOIMNNE53LN4MBSDDL65NQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:28.083140990 (551h16m27.954224025s) [noise=true , address=145.79.227.44:4212 , piece_id=V73IIIU7CCTULBPEXD2YWSFUUGEP2ETVL7LQGIWWEXICZ5SK4BQA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:28.129315084 (551h16m28.000398119s) [noise=true , address=145.79.227.40:4075 , piece_id=XKJZ2GSNELWKMTMZA5EZ2YTM62XREZQDFI7MDDKAPE3YXRY4KWNA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:28.129340211 (551h16m28.000423246s) [seg_pos={0 0}, upload=483348]: Segment reached optimal threshold of 14 pieces.
Dec 28 14:45:28.129532938 (551h16m28.000615973s) [noise=true , address=77.90.31.21:4889 , piece_id=WSIKD7GCMKHWEFYC7UZ3YWGQHQDIZVW2YJY4BD5ENMCYBJZJMXEQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wcX8iEhi5LrftcXUQE3Lrm4igNwhDtvT3RNBERTGsKyQ6KLj6): protocol: expected piece hash; context canceled; EOF
Dec 28 14:45:28.129539380 (551h16m28.000622425s) [noise=true , address=77.90.31.21:4889 , piece_id=WSIKD7GCMKHWEFYC7UZ3YWGQHQDIZVW2YJY4BD5ENMCYBJZJMXEQ, seg_pos={0 0}, upload=483348]: Long tail cancellation due to optimal threshold
Dec 28 14:45:28.129666382 (551h16m28.000749418s) [noise=true , address=64.49.13.169:4711 , piece_id=YHY2V4EHCMQEDR6OTZE2POM45Y7FYVNOSEGSUOM5IYWCDWDLRIKQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:47258->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:47258->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47258->64.49.13.169:4711: use of closed network connection
Dec 28 14:45:28.129671853 (551h16m28.000754898s) [noise=true , address=64.49.13.169:4711 , piece_id=YHY2V4EHCMQEDR6OTZE2POM45Y7FYVNOSEGSUOM5IYWCDWDLRIKQ, seg_pos={0 0}, upload=483348]: Long tail cancellation due to optimal threshold
Dec 28 14:45:28.129713071 (551h16m28.000796116s) [noise=true , address=77.90.31.14:4688 , piece_id=LNETNIH6PNQ2OR2NQKBWWWOJEFYXIGYHHH34WW7DHTKKYQWJ6DLA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz): protocol: expected piece hash; context canceled; EOF
Dec 28 14:45:28.129770090 (551h16m28.000853125s) [noise=true , address=77.90.31.14:4688 , piece_id=LNETNIH6PNQ2OR2NQKBWWWOJEFYXIGYHHH34WW7DHTKKYQWJ6DLA, seg_pos={0 0}, upload=483348]: Long tail cancellation due to optimal threshold
Dec 28 14:45:28.129862666 (551h16m28.000945711s) [noise=true , address=193.149.164.143:4697 , piece_id=6SHPKJO6AZLKYPIQWJWNI4YSAMLAVU7BGDGQLQC4I6AGCRWT52DA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ): protocol: expected piece hash; context canceled; EOF
Dec 28 14:45:28.129870651 (551h16m28.000953686s) [noise=true , address=193.149.164.143:4697 , piece_id=6SHPKJO6AZLKYPIQWJWNI4YSAMLAVU7BGDGQLQC4I6AGCRWT52DA, seg_pos={0 0}, upload=483348]: Long tail cancellation due to optimal threshold
Dec 28 14:45:28.129970341 (551h16m28.001053376s) [seg_pos={0 0}, upload=483348]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 14:45:28.130053620 (551h16m28.001136665s) [upload=483348]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 14:45:28.130060553 (551h16m28.001143588s) [upload=483348]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 14:45:28.417819022 (551h16m28.288902057s) [upload=483348]: Done uploading segments...
Dec 28 14:45:28.417844872 (551h16m28.288927927s) [upload=483348]: Upload finished. err: <nil>
Duration: 54.623216954s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
9.330310076s trace:
https://jaeger.dev.storj.io/trace/35a5d4335e2cde85
FAIL (2025-12-28 14:56:47 UTC)
orbiter warning: context deadline exceeded
Duration: 9.330336716s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
20 out of 60 is failed (threshold=3) [XX.XX.X.X..XXXXX....X...X..X..X...X...X.X.........X.....X...] duration:
5.825546642s trace:
https://jaeger.dev.storj.io/trace/400f8ae82c545e27
FAIL (2025-12-28 14:56:35 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: P3XVLM2MNM224KOHB65XZBQNGQNOVJ37RY2FR4JSPRZPOFZM5W4Q) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: CG7G54I3IGFS32ZAY2WJHDRCYAGLFVO6TOVKX7R4CUJ6UO7V3G4Q) context deadline exceeded
======LOGS======
28 14:54:28.651637995 (551h25m28.52272104s) [upload=483478]: Starting upload
Dec 28 14:54:28.651775788 (551h25m28.522858823s) [upload=483478]: Done starting upload
Dec 28 14:54:28.651857663 (551h25m28.522940698s) [upload=483478]: Uploading segments...
Dec 28 14:54:28.651966430 (551h25m28.523049475s) [upload=483478]: Got next segment. Inline: false
Dec 28 14:54:28.652001437 (551h25m28.523084472s) [upload=483478]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 14:54:28.652006597 (551h25m28.523089632s) [upload=483478]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 14:54:28.897465992 (551h25m28.768549038s) [seg_pos={0 0}, upload=483478]: Begin upload segment...
Dec 28 14:54:28.897524775 (551h25m28.76860782s) [seg_pos={0 0}, upload=483478]: Done begin upload segment.
Dec 28 14:54:28.897584187 (551h25m28.768667232s) [noise=true , address=145.79.227.50:4373 , piece_id=F5HNDNMLFF4GW3EVG2DR2V2I7D6OZN7NA7B6WZMKWGTSZ5FPKREA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.897595700 (551h25m28.768678735s) [noise=true , address=45.91.116.7:4010 , piece_id=P3XVLM2MNM224KOHB65XZBQNGQNOVJ37RY2FR4JSPRZPOFZM5W4Q, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.898192205 (551h25m28.76927524s) [noise=true , address=45.91.116.6:4973 , piece_id=ZCNSUED6MUMN4OCGGVXD3RVV6OI3D3DRDRLD43YGE2WVDDTGW3FQ, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.898681176 (551h25m28.769764212s) [noise=true , address=145.79.227.18:4007 , piece_id=Y52XM3XDSZFKHR6C7CYQ7VIUQGMWNB3JQMGPRQUVKENFJ7RRSQYQ, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.899027356 (551h25m28.770110391s) [noise=true , address=145.79.227.21:4473 , piece_id=44NH6BNBDRVJMYAJDGHIIWRPHQZGHCA3SIUGQ4IHTAYSMW3U2ESA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.899416557 (551h25m28.770499602s) [noise=true , address=145.79.227.5:4223 , piece_id=T6JVT2OXKEBVYP4ICGLCPK3CLN5H25TFTRXC22E7OBD7NLA3HDLA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.899745403 (551h25m28.770828448s) [noise=true , address=45.91.116.5:4298 , piece_id=F6UV3BDNWFAGWBBJLNSBQETPARFEGKB4Z6POFKVBV6X5DNIFOODA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.900154752 (551h25m28.771237797s) [noise=true , address=145.79.227.44:4194 , piece_id=77E4L3WGPLKI2HUTR6UAOHYTVUMBBGMGVALT6R42L7PLMVIT7JJQ, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.900476626 (551h25m28.771559661s) [noise=true , address=145.79.227.15:4874 , piece_id=3CYMIHKUTOXHXJBVJKUUJC6HKTQA4AD5HCET3UJNDMFHGXGXIOOA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.902303083 (551h25m28.773386118s) [noise=true , address=145.79.227.36:4519 , piece_id=I7VEEU6WTZG4YRMJIKUQTWLPFBECYVYTV4Z43NP3U2QGHPGB3EEA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.903509890 (551h25m28.774592926s) [noise=true , address=77.90.31.13:4626 , piece_id=CG7G54I3IGFS32ZAY2WJHDRCYAGLFVO6TOVKX7R4CUJ6UO7V3G4Q, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.905006530 (551h25m28.776089565s) [noise=true , address=77.90.31.16:4735 , piece_id=Z2JJ7IIWX3RQLXM3HABIULPLEQSMDKVXC6F7YJCW3HCVEBRRYM6Q, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.906620755 (551h25m28.77770379s) [noise=true , address=145.79.227.52:4445 , piece_id=ZKUYH45CZIQNCWP5Q3JMEJTOA5TJSTZL7TTWOBGF5QOP245ZNHHQ, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.908401716 (551h25m28.779484761s) [noise=true , address=145.79.227.2:4113 , piece_id=PIBB5A34I4C3T5RX4DPHMSFBGJGZDGWOP7XBELPTOYA45DRVZZCA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.909977536 (551h25m28.781060571s) [noise=true , address=145.79.227.32:4835 , piece_id=FVGQX4522G64FUDBCKN6LXSTECVOXZONMO3WPUQGWYUMKKLHAL7Q, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.912006570 (551h25m28.783089625s) [noise=true , address=145.79.227.10:4401 , piece_id=A4UVXDWW66OY2DIFDGN47XYGK255NTLGR2U36NUDF32AOJD2LFKA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:30.616536389 (551h25m30.487619424s) [upload=483478]: Next returned nil segment
Dec 28 14:54:30.616555134 (551h25m30.487638179s) [upload=483478]: Waiting for error group managing segments...
Dec 28 14:54:30.902690564 (551h25m30.773773619s) [noise=true , address=145.79.227.5:4223 , piece_id=T6JVT2OXKEBVYP4ICGLCPK3CLN5H25TFTRXC22E7OBD7NLA3HDLA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.908186579 (551h25m30.779269624s) [noise=true , address=145.79.227.2:4113 , piece_id=PIBB5A34I4C3T5RX4DPHMSFBGJGZDGWOP7XBELPTOYA45DRVZZCA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.908476522 (551h25m30.779559557s) [noise=true , address=145.79.227.21:4473 , piece_id=44NH6BNBDRVJMYAJDGHIIWRPHQZGHCA3SIUGQ4IHTAYSMW3U2ESA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.910250058 (551h25m30.781333113s) [noise=true , address=145.79.227.18:4007 , piece_id=Y52XM3XDSZFKHR6C7CYQ7VIUQGMWNB3JQMGPRQUVKENFJ7RRSQYQ, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.911106399 (551h25m30.782189444s) [noise=true , address=145.79.227.50:4373 , piece_id=F5HNDNMLFF4GW3EVG2DR2V2I7D6OZN7NA7B6WZMKWGTSZ5FPKREA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.914846541 (551h25m30.785929587s) [noise=true , address=145.79.227.10:4401 , piece_id=A4UVXDWW66OY2DIFDGN47XYGK255NTLGR2U36NUDF32AOJD2LFKA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.919475566 (551h25m30.790558621s) [noise=true , address=145.79.227.52:4445 , piece_id=ZKUYH45CZIQNCWP5Q3JMEJTOA5TJSTZL7TTWOBGF5QOP245ZNHHQ, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.921006410 (551h25m30.792089455s) [noise=true , address=145.79.227.36:4519 , piece_id=I7VEEU6WTZG4YRMJIKUQTWLPFBECYVYTV4Z43NP3U2QGHPGB3EEA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.921040545 (551h25m30.7921236s) [noise=true , address=145.79.227.15:4874 , piece_id=3CYMIHKUTOXHXJBVJKUUJC6HKTQA4AD5HCET3UJNDMFHGXGXIOOA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.941624211 (551h25m30.812707256s) [noise=true , address=145.79.227.32:4835 , piece_id=FVGQX4522G64FUDBCKN6LXSTECVOXZONMO3WPUQGWYUMKKLHAL7Q, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:31.182363167 (551h25m31.053446202s) [noise=true , address=77.90.31.16:4735 , piece_id=Z2JJ7IIWX3RQLXM3HABIULPLEQSMDKVXC6F7YJCW3HCVEBRRYM6Q, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:31.261978689 (551h25m31.133061734s) [noise=true , address=77.90.31.13:4626 , piece_id=CG7G54I3IGFS32ZAY2WJHDRCYAGLFVO6TOVKX7R4CUJ6UO7V3G4Q, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:31.276375082 (551h25m31.147458117s) [noise=true , address=45.91.116.7:4010 , piece_id=P3XVLM2MNM224KOHB65XZBQNGQNOVJ37RY2FR4JSPRZPOFZM5W4Q, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:31.276394178 (551h25m31.147477223s) [seg_pos={0 0}, upload=483478]: Segment reached optimal threshold of 13 pieces.
Dec 28 14:54:31.276572508 (551h25m31.147655543s) [noise=true , address=45.91.116.6:4973 , piece_id=ZCNSUED6MUMN4OCGGVXD3RVV6OI3D3DRDRLD43YGE2WVDDTGW3FQ, seg_pos={0 0}, upload=483478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF
Dec 28 14:54:31.276578930 (551h25m31.147661975s) [noise=true , address=45.91.116.6:4973 , piece_id=ZCNSUED6MUMN4OCGGVXD3RVV6OI3D3DRDRLD43YGE2WVDDTGW3FQ, seg_pos={0 0}, upload=483478]: Long tail cancellation due to optimal threshold
Dec 28 14:54:31.276703968 (551h25m31.147787003s) [noise=true , address=45.91.116.5:4298 , piece_id=F6UV3BDNWFAGWBBJLNSBQETPARFEGKB4Z6POFKVBV6X5DNIFOODA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:34120->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:34120->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34120->45.91.116.5:4298: use of closed network connection
Dec 28 14:54:31.276709198 (551h25m31.147792233s) [noise=true , address=45.91.116.5:4298 , piece_id=F6UV3BDNWFAGWBBJLNSBQETPARFEGKB4Z6POFKVBV6X5DNIFOODA, seg_pos={0 0}, upload=483478]: Long tail cancellation due to optimal threshold
Dec 28 14:54:31.277040298 (551h25m31.148123343s) [noise=true , address=145.79.227.44:4194 , piece_id=77E4L3WGPLKI2HUTR6UAOHYTVUMBBGMGVALT6R42L7PLMVIT7JJQ, seg_pos={0 0}, upload=483478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF
Dec 28 14:54:31.277060938 (551h25m31.148143983s) [noise=true , address=145.79.227.44:4194 , piece_id=77E4L3WGPLKI2HUTR6UAOHYTVUMBBGMGVALT6R42L7PLMVIT7JJQ, seg_pos={0 0}, upload=483478]: Long tail cancellation due to optimal threshold
Dec 28 14:54:31.277105372 (551h25m31.148188417s) [seg_pos={0 0}, upload=483478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 14:54:31.277258323 (551h25m31.148341378s) [upload=483478]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 14:54:31.277265838 (551h25m31.148348903s) [upload=483478]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 14:54:31.575778626 (551h25m31.446861691s) [upload=483478]: Done uploading segments...
Dec 28 14:54:31.575803704 (551h25m31.446886749s) [upload=483478]: Upload finished. err: <nil>
Duration: 5.825572602s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [X....................X...........X..........................] duration:
14.94163699s trace:
https://jaeger.dev.storj.io/trace/70ba9ccd98ccb34e
FAIL (2025-12-28 14:56:35 UTC)
orbiter warning: context deadline exceeded
Duration: 14.941753693s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
10.601502772s trace:
https://jaeger.dev.storj.io/trace/35a5d4335e2cde85
FAIL (2025-12-28 14:56:32 UTC)
orbiter warning: context deadline exceeded
Duration: 10.601529883s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
48 out of 60 is failed (threshold=6) [X.XXXXXXXXXXXXXXXXXXXXXXX.X.XX.XXXXXXXX.X..XXXXX.X.X.XXXXX..] duration:
5.000640303s trace:
https://jaeger.dev.storj.io/trace/131df032dcbcf2b
FAIL (2025-12-28 14:56:20 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000658418s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
20 out of 60 is failed (threshold=3) [.XX.XX.X.X..XXXXX....X...X..X..X...X...X.X.........X.....X..] duration:
6.085860922s trace:
https://jaeger.dev.storj.io/trace/400f8ae82c545e27
FAIL (2025-12-28 14:55:57 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: P3XVLM2MNM224KOHB65XZBQNGQNOVJ37RY2FR4JSPRZPOFZM5W4Q) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: CG7G54I3IGFS32ZAY2WJHDRCYAGLFVO6TOVKX7R4CUJ6UO7V3G4Q) context deadline exceeded
======LOGS======
28 14:54:28.651637995 (551h25m28.52272104s) [upload=483478]: Starting upload
Dec 28 14:54:28.651775788 (551h25m28.522858823s) [upload=483478]: Done starting upload
Dec 28 14:54:28.651857663 (551h25m28.522940698s) [upload=483478]: Uploading segments...
Dec 28 14:54:28.651966430 (551h25m28.523049475s) [upload=483478]: Got next segment. Inline: false
Dec 28 14:54:28.652001437 (551h25m28.523084472s) [upload=483478]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 14:54:28.652006597 (551h25m28.523089632s) [upload=483478]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 14:54:28.897465992 (551h25m28.768549038s) [seg_pos={0 0}, upload=483478]: Begin upload segment...
Dec 28 14:54:28.897524775 (551h25m28.76860782s) [seg_pos={0 0}, upload=483478]: Done begin upload segment.
Dec 28 14:54:28.897584187 (551h25m28.768667232s) [noise=true , address=145.79.227.50:4373 , piece_id=F5HNDNMLFF4GW3EVG2DR2V2I7D6OZN7NA7B6WZMKWGTSZ5FPKREA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.897595700 (551h25m28.768678735s) [noise=true , address=45.91.116.7:4010 , piece_id=P3XVLM2MNM224KOHB65XZBQNGQNOVJ37RY2FR4JSPRZPOFZM5W4Q, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.898192205 (551h25m28.76927524s) [noise=true , address=45.91.116.6:4973 , piece_id=ZCNSUED6MUMN4OCGGVXD3RVV6OI3D3DRDRLD43YGE2WVDDTGW3FQ, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.898681176 (551h25m28.769764212s) [noise=true , address=145.79.227.18:4007 , piece_id=Y52XM3XDSZFKHR6C7CYQ7VIUQGMWNB3JQMGPRQUVKENFJ7RRSQYQ, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.899027356 (551h25m28.770110391s) [noise=true , address=145.79.227.21:4473 , piece_id=44NH6BNBDRVJMYAJDGHIIWRPHQZGHCA3SIUGQ4IHTAYSMW3U2ESA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.899416557 (551h25m28.770499602s) [noise=true , address=145.79.227.5:4223 , piece_id=T6JVT2OXKEBVYP4ICGLCPK3CLN5H25TFTRXC22E7OBD7NLA3HDLA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.899745403 (551h25m28.770828448s) [noise=true , address=45.91.116.5:4298 , piece_id=F6UV3BDNWFAGWBBJLNSBQETPARFEGKB4Z6POFKVBV6X5DNIFOODA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.900154752 (551h25m28.771237797s) [noise=true , address=145.79.227.44:4194 , piece_id=77E4L3WGPLKI2HUTR6UAOHYTVUMBBGMGVALT6R42L7PLMVIT7JJQ, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.900476626 (551h25m28.771559661s) [noise=true , address=145.79.227.15:4874 , piece_id=3CYMIHKUTOXHXJBVJKUUJC6HKTQA4AD5HCET3UJNDMFHGXGXIOOA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.902303083 (551h25m28.773386118s) [noise=true , address=145.79.227.36:4519 , piece_id=I7VEEU6WTZG4YRMJIKUQTWLPFBECYVYTV4Z43NP3U2QGHPGB3EEA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.903509890 (551h25m28.774592926s) [noise=true , address=77.90.31.13:4626 , piece_id=CG7G54I3IGFS32ZAY2WJHDRCYAGLFVO6TOVKX7R4CUJ6UO7V3G4Q, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.905006530 (551h25m28.776089565s) [noise=true , address=77.90.31.16:4735 , piece_id=Z2JJ7IIWX3RQLXM3HABIULPLEQSMDKVXC6F7YJCW3HCVEBRRYM6Q, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.906620755 (551h25m28.77770379s) [noise=true , address=145.79.227.52:4445 , piece_id=ZKUYH45CZIQNCWP5Q3JMEJTOA5TJSTZL7TTWOBGF5QOP245ZNHHQ, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.908401716 (551h25m28.779484761s) [noise=true , address=145.79.227.2:4113 , piece_id=PIBB5A34I4C3T5RX4DPHMSFBGJGZDGWOP7XBELPTOYA45DRVZZCA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.909977536 (551h25m28.781060571s) [noise=true , address=145.79.227.32:4835 , piece_id=FVGQX4522G64FUDBCKN6LXSTECVOXZONMO3WPUQGWYUMKKLHAL7Q, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.912006570 (551h25m28.783089625s) [noise=true , address=145.79.227.10:4401 , piece_id=A4UVXDWW66OY2DIFDGN47XYGK255NTLGR2U36NUDF32AOJD2LFKA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:30.616536389 (551h25m30.487619424s) [upload=483478]: Next returned nil segment
Dec 28 14:54:30.616555134 (551h25m30.487638179s) [upload=483478]: Waiting for error group managing segments...
Dec 28 14:54:30.902690564 (551h25m30.773773619s) [noise=true , address=145.79.227.5:4223 , piece_id=T6JVT2OXKEBVYP4ICGLCPK3CLN5H25TFTRXC22E7OBD7NLA3HDLA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.908186579 (551h25m30.779269624s) [noise=true , address=145.79.227.2:4113 , piece_id=PIBB5A34I4C3T5RX4DPHMSFBGJGZDGWOP7XBELPTOYA45DRVZZCA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.908476522 (551h25m30.779559557s) [noise=true , address=145.79.227.21:4473 , piece_id=44NH6BNBDRVJMYAJDGHIIWRPHQZGHCA3SIUGQ4IHTAYSMW3U2ESA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.910250058 (551h25m30.781333113s) [noise=true , address=145.79.227.18:4007 , piece_id=Y52XM3XDSZFKHR6C7CYQ7VIUQGMWNB3JQMGPRQUVKENFJ7RRSQYQ, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.911106399 (551h25m30.782189444s) [noise=true , address=145.79.227.50:4373 , piece_id=F5HNDNMLFF4GW3EVG2DR2V2I7D6OZN7NA7B6WZMKWGTSZ5FPKREA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.914846541 (551h25m30.785929587s) [noise=true , address=145.79.227.10:4401 , piece_id=A4UVXDWW66OY2DIFDGN47XYGK255NTLGR2U36NUDF32AOJD2LFKA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.919475566 (551h25m30.790558621s) [noise=true , address=145.79.227.52:4445 , piece_id=ZKUYH45CZIQNCWP5Q3JMEJTOA5TJSTZL7TTWOBGF5QOP245ZNHHQ, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.921006410 (551h25m30.792089455s) [noise=true , address=145.79.227.36:4519 , piece_id=I7VEEU6WTZG4YRMJIKUQTWLPFBECYVYTV4Z43NP3U2QGHPGB3EEA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.921040545 (551h25m30.7921236s) [noise=true , address=145.79.227.15:4874 , piece_id=3CYMIHKUTOXHXJBVJKUUJC6HKTQA4AD5HCET3UJNDMFHGXGXIOOA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.941624211 (551h25m30.812707256s) [noise=true , address=145.79.227.32:4835 , piece_id=FVGQX4522G64FUDBCKN6LXSTECVOXZONMO3WPUQGWYUMKKLHAL7Q, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:31.182363167 (551h25m31.053446202s) [noise=true , address=77.90.31.16:4735 , piece_id=Z2JJ7IIWX3RQLXM3HABIULPLEQSMDKVXC6F7YJCW3HCVEBRRYM6Q, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:31.261978689 (551h25m31.133061734s) [noise=true , address=77.90.31.13:4626 , piece_id=CG7G54I3IGFS32ZAY2WJHDRCYAGLFVO6TOVKX7R4CUJ6UO7V3G4Q, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:31.276375082 (551h25m31.147458117s) [noise=true , address=45.91.116.7:4010 , piece_id=P3XVLM2MNM224KOHB65XZBQNGQNOVJ37RY2FR4JSPRZPOFZM5W4Q, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:31.276394178 (551h25m31.147477223s) [seg_pos={0 0}, upload=483478]: Segment reached optimal threshold of 13 pieces.
Dec 28 14:54:31.276572508 (551h25m31.147655543s) [noise=true , address=45.91.116.6:4973 , piece_id=ZCNSUED6MUMN4OCGGVXD3RVV6OI3D3DRDRLD43YGE2WVDDTGW3FQ, seg_pos={0 0}, upload=483478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF
Dec 28 14:54:31.276578930 (551h25m31.147661975s) [noise=true , address=45.91.116.6:4973 , piece_id=ZCNSUED6MUMN4OCGGVXD3RVV6OI3D3DRDRLD43YGE2WVDDTGW3FQ, seg_pos={0 0}, upload=483478]: Long tail cancellation due to optimal threshold
Dec 28 14:54:31.276703968 (551h25m31.147787003s) [noise=true , address=45.91.116.5:4298 , piece_id=F6UV3BDNWFAGWBBJLNSBQETPARFEGKB4Z6POFKVBV6X5DNIFOODA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:34120->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:34120->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34120->45.91.116.5:4298: use of closed network connection
Dec 28 14:54:31.276709198 (551h25m31.147792233s) [noise=true , address=45.91.116.5:4298 , piece_id=F6UV3BDNWFAGWBBJLNSBQETPARFEGKB4Z6POFKVBV6X5DNIFOODA, seg_pos={0 0}, upload=483478]: Long tail cancellation due to optimal threshold
Dec 28 14:54:31.277040298 (551h25m31.148123343s) [noise=true , address=145.79.227.44:4194 , piece_id=77E4L3WGPLKI2HUTR6UAOHYTVUMBBGMGVALT6R42L7PLMVIT7JJQ, seg_pos={0 0}, upload=483478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF
Dec 28 14:54:31.277060938 (551h25m31.148143983s) [noise=true , address=145.79.227.44:4194 , piece_id=77E4L3WGPLKI2HUTR6UAOHYTVUMBBGMGVALT6R42L7PLMVIT7JJQ, seg_pos={0 0}, upload=483478]: Long tail cancellation due to optimal threshold
Dec 28 14:54:31.277105372 (551h25m31.148188417s) [seg_pos={0 0}, upload=483478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 14:54:31.277258323 (551h25m31.148341378s) [upload=483478]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 14:54:31.277265838 (551h25m31.148348903s) [upload=483478]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 14:54:31.575778626 (551h25m31.446861691s) [upload=483478]: Done uploading segments...
Dec 28 14:54:31.575803704 (551h25m31.446886749s) [upload=483478]: Upload finished. err: <nil>
Duration: 6.085891018s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [....X...X............X...............X......X....X..........] duration:
55.514633917s trace:
https://jaeger.dev.storj.io/trace/1440a926e26dd799
FAIL (2025-12-28 14:55:53 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
noise=true , address=145.79.227.44:4212 , piece_id=V73IIIU7CCTULBPEXD2YWSFUUGEP2ETVL7LQGIWWEXICZ5SK4BQA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.239195729 (551h16m25.110278774s) [noise=true , address=145.79.227.34:4931 , piece_id=IBNDRK5G5MGWQKRTJ3CS7LI7F3TFUJDHZDG6HJZ44XOQAEWBVEBA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.239549302 (551h16m25.110632347s) [noise=true , address=64.49.13.140:4515 , piece_id=42YNEU3HOVPRPUVOCPKYSQUBTDLBQELQCWNIYJBDXD2RV2NRDJDQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.239966557 (551h16m25.111049602s) [noise=true , address=77.90.31.18:4816 , piece_id=J3DJPEVEESV6EA5KJU6TN7MO4ATQZDBQOSJSH5S5KSZRYGHHZRSQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.240145648 (551h16m25.111228683s) [noise=true , address=77.90.31.14:4688 , piece_id=LNETNIH6PNQ2OR2NQKBWWWOJEFYXIGYHHH34WW7DHTKKYQWJ6DLA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.240933478 (551h16m25.112016523s) [noise=true , address=77.90.31.19:4867 , piece_id=NWQD7PXUGDHUKRF4J3E6EUORCKUM3TDAD2EJH7F3CE55G4EA7SJQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.241389286 (551h16m25.112472331s) [noise=true , address=145.79.227.40:4075 , piece_id=XKJZ2GSNELWKMTMZA5EZ2YTM62XREZQDFI7MDDKAPE3YXRY4KWNA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.241451415 (551h16m25.11253445s) [noise=true , address=77.90.31.21:4889 , piece_id=WSIKD7GCMKHWEFYC7UZ3YWGQHQDIZVW2YJY4BD5ENMCYBJZJMXEQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.241843531 (551h16m25.112926566s) [noise=true , address=145.79.227.46:4608 , piece_id=T5G4T4OU62UPXPIACXJQEZ36O2N63HL4L4HRMW3EN6B4PIE3M7YA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.241911240 (551h16m25.112994285s) [noise=true , address=64.49.13.169:4711 , piece_id=YHY2V4EHCMQEDR6OTZE2POM45Y7FYVNOSEGSUOM5IYWCDWDLRIKQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.242196533 (551h16m25.113279578s) [noise=true , address=193.149.164.143:4697 , piece_id=6SHPKJO6AZLKYPIQWJWNI4YSAMLAVU7BGDGQLQC4I6AGCRWT52DA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.242532343 (551h16m25.113615378s) [noise=true , address=145.79.227.29:4947 , piece_id=FSJZQVSKN76O5RLIVQDPKAN2PDV4VLRCMBRA5EYJGFOETOZ6D7AQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.242582478 (551h16m25.113665613s) [noise=true , address=145.79.227.18:4010 , piece_id=GVY67LYRK3UZRX6OLLWC5JCJEGV3RCIS6DATIDMQCBJSL7DCAE4Q, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.242986888 (551h16m25.114069923s) [noise=true , address=64.49.13.27:4966 , piece_id=N3R7JO5PL47V7ZUXNLXFOHSI2MEJ2SVQNHE6M2DWDYF3D4L34ZCA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.243048475 (551h16m25.11413151s) [noise=true , address=77.90.31.11:4585 , piece_id=5MKOQOD3O7TXKXODJWTVZI5EAKBEO7JE3KKXD63F6SO3EBBSVD6A, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.243336464 (551h16m25.114419499s) [noise=true , address=77.90.31.17:4801 , piece_id=OWQPIGFIL5SFLUI77IYA3LKVUFPSBL6JSJO6XWZLQFV7GXHF3QOA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:26.537548687 (551h16m26.408631722s) [upload=483348]: Next returned nil segment
Dec 28 14:45:26.537559538 (551h16m26.408642573s) [upload=483348]: Waiting for error group managing segments...
Dec 28 14:45:26.899924119 (551h16m26.771007164s) [noise=true , address=64.49.13.140:4515 , piece_id=42YNEU3HOVPRPUVOCPKYSQUBTDLBQELQCWNIYJBDXD2RV2NRDJDQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:26.977957828 (551h16m26.849040873s) [noise=true , address=64.49.13.27:4966 , piece_id=N3R7JO5PL47V7ZUXNLXFOHSI2MEJ2SVQNHE6M2DWDYF3D4L34ZCA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.120995816 (551h16m26.992078862s) [noise=true , address=145.79.227.46:4608 , piece_id=T5G4T4OU62UPXPIACXJQEZ36O2N63HL4L4HRMW3EN6B4PIE3M7YA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.151904071 (551h16m27.022987116s) [noise=true , address=145.79.227.34:4931 , piece_id=IBNDRK5G5MGWQKRTJ3CS7LI7F3TFUJDHZDG6HJZ44XOQAEWBVEBA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.165435146 (551h16m27.036518181s) [noise=true , address=145.79.227.29:4947 , piece_id=FSJZQVSKN76O5RLIVQDPKAN2PDV4VLRCMBRA5EYJGFOETOZ6D7AQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.175130708 (551h16m27.046213754s) [noise=true , address=145.79.227.18:4010 , piece_id=GVY67LYRK3UZRX6OLLWC5JCJEGV3RCIS6DATIDMQCBJSL7DCAE4Q, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.352651587 (551h16m27.223734632s) [noise=true , address=77.90.31.19:4867 , piece_id=NWQD7PXUGDHUKRF4J3E6EUORCKUM3TDAD2EJH7F3CE55G4EA7SJQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.375485928 (551h16m27.246568973s) [noise=true , address=77.90.31.11:4585 , piece_id=5MKOQOD3O7TXKXODJWTVZI5EAKBEO7JE3KKXD63F6SO3EBBSVD6A, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.428947778 (551h16m27.300030823s) [noise=true , address=77.90.31.18:4816 , piece_id=J3DJPEVEESV6EA5KJU6TN7MO4ATQZDBQOSJSH5S5KSZRYGHHZRSQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.567872934 (551h16m27.438955969s) [noise=true , address=77.90.31.17:4801 , piece_id=OWQPIGFIL5SFLUI77IYA3LKVUFPSBL6JSJO6XWZLQFV7GXHF3QOA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.983670843 (551h16m27.854753878s) [noise=true , address=77.90.31.13:4650 , piece_id=NHTWFNKCRHELW7BJ2NR4GGHY4LU743RVLCCR236AO4MQUZD7KMEQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:28.019933813 (551h16m27.891016848s) [noise=true , address=145.79.227.30:4794 , piece_id=OZXLKN4V62NRD3WCMF65C7U7OPM2FPBOIMNNE53LN4MBSDDL65NQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:28.083140990 (551h16m27.954224025s) [noise=true , address=145.79.227.44:4212 , piece_id=V73IIIU7CCTULBPEXD2YWSFUUGEP2ETVL7LQGIWWEXICZ5SK4BQA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:28.129315084 (551h16m28.000398119s) [noise=true , address=145.79.227.40:4075 , piece_id=XKJZ2GSNELWKMTMZA5EZ2YTM62XREZQDFI7MDDKAPE3YXRY4KWNA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:28.129340211 (551h16m28.000423246s) [seg_pos={0 0}, upload=483348]: Segment reached optimal threshold of 14 pieces.
Dec 28 14:45:28.129532938 (551h16m28.000615973s) [noise=true , address=77.90.31.21:4889 , piece_id=WSIKD7GCMKHWEFYC7UZ3YWGQHQDIZVW2YJY4BD5ENMCYBJZJMXEQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wcX8iEhi5LrftcXUQE3Lrm4igNwhDtvT3RNBERTGsKyQ6KLj6): protocol: expected piece hash; context canceled; EOF
Dec 28 14:45:28.129539380 (551h16m28.000622425s) [noise=true , address=77.90.31.21:4889 , piece_id=WSIKD7GCMKHWEFYC7UZ3YWGQHQDIZVW2YJY4BD5ENMCYBJZJMXEQ, seg_pos={0 0}, upload=483348]: Long tail cancellation due to optimal threshold
Dec 28 14:45:28.129666382 (551h16m28.000749418s) [noise=true , address=64.49.13.169:4711 , piece_id=YHY2V4EHCMQEDR6OTZE2POM45Y7FYVNOSEGSUOM5IYWCDWDLRIKQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:47258->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:47258->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47258->64.49.13.169:4711: use of closed network connection
Dec 28 14:45:28.129671853 (551h16m28.000754898s) [noise=true , address=64.49.13.169:4711 , piece_id=YHY2V4EHCMQEDR6OTZE2POM45Y7FYVNOSEGSUOM5IYWCDWDLRIKQ, seg_pos={0 0}, upload=483348]: Long tail cancellation due to optimal threshold
Dec 28 14:45:28.129713071 (551h16m28.000796116s) [noise=true , address=77.90.31.14:4688 , piece_id=LNETNIH6PNQ2OR2NQKBWWWOJEFYXIGYHHH34WW7DHTKKYQWJ6DLA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz): protocol: expected piece hash; context canceled; EOF
Dec 28 14:45:28.129770090 (551h16m28.000853125s) [noise=true , address=77.90.31.14:4688 , piece_id=LNETNIH6PNQ2OR2NQKBWWWOJEFYXIGYHHH34WW7DHTKKYQWJ6DLA, seg_pos={0 0}, upload=483348]: Long tail cancellation due to optimal threshold
Dec 28 14:45:28.129862666 (551h16m28.000945711s) [noise=true , address=193.149.164.143:4697 , piece_id=6SHPKJO6AZLKYPIQWJWNI4YSAMLAVU7BGDGQLQC4I6AGCRWT52DA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ): protocol: expected piece hash; context canceled; EOF
Dec 28 14:45:28.129870651 (551h16m28.000953686s) [noise=true , address=193.149.164.143:4697 , piece_id=6SHPKJO6AZLKYPIQWJWNI4YSAMLAVU7BGDGQLQC4I6AGCRWT52DA, seg_pos={0 0}, upload=483348]: Long tail cancellation due to optimal threshold
Dec 28 14:45:28.129970341 (551h16m28.001053376s) [seg_pos={0 0}, upload=483348]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 14:45:28.130053620 (551h16m28.001136665s) [upload=483348]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 14:45:28.130060553 (551h16m28.001143588s) [upload=483348]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 14:45:28.417819022 (551h16m28.288902057s) [upload=483348]: Done uploading segments...
Dec 28 14:45:28.417844872 (551h16m28.288927927s) [upload=483348]: Upload finished. err: <nil>
Duration: 55.514664916s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
48 out of 60 is failed (threshold=6) [.X.XXXXXXXXXXXXXXXXXXXXXXX.X.XX.XXXXXXXX.X..XXXXX.X.X.XXXXX.] duration:
3.544103916s trace:
https://jaeger.dev.storj.io/trace/131df032dcbcf2b
FAIL (2025-12-28 14:55:48 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.544131447s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
20 out of 60 is failed (threshold=3) [..XX.XX.X.X..XXXXX....X...X..X..X...X...X.X.........X.....X.] duration:
9.173072819s trace:
https://jaeger.dev.storj.io/trace/400f8ae82c545e27
FAIL (2025-12-28 14:55:44 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: P3XVLM2MNM224KOHB65XZBQNGQNOVJ37RY2FR4JSPRZPOFZM5W4Q) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: CG7G54I3IGFS32ZAY2WJHDRCYAGLFVO6TOVKX7R4CUJ6UO7V3G4Q) context deadline exceeded
======LOGS======
28 14:54:28.651637995 (551h25m28.52272104s) [upload=483478]: Starting upload
Dec 28 14:54:28.651775788 (551h25m28.522858823s) [upload=483478]: Done starting upload
Dec 28 14:54:28.651857663 (551h25m28.522940698s) [upload=483478]: Uploading segments...
Dec 28 14:54:28.651966430 (551h25m28.523049475s) [upload=483478]: Got next segment. Inline: false
Dec 28 14:54:28.652001437 (551h25m28.523084472s) [upload=483478]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 14:54:28.652006597 (551h25m28.523089632s) [upload=483478]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 14:54:28.897465992 (551h25m28.768549038s) [seg_pos={0 0}, upload=483478]: Begin upload segment...
Dec 28 14:54:28.897524775 (551h25m28.76860782s) [seg_pos={0 0}, upload=483478]: Done begin upload segment.
Dec 28 14:54:28.897584187 (551h25m28.768667232s) [noise=true , address=145.79.227.50:4373 , piece_id=F5HNDNMLFF4GW3EVG2DR2V2I7D6OZN7NA7B6WZMKWGTSZ5FPKREA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.897595700 (551h25m28.768678735s) [noise=true , address=45.91.116.7:4010 , piece_id=P3XVLM2MNM224KOHB65XZBQNGQNOVJ37RY2FR4JSPRZPOFZM5W4Q, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.898192205 (551h25m28.76927524s) [noise=true , address=45.91.116.6:4973 , piece_id=ZCNSUED6MUMN4OCGGVXD3RVV6OI3D3DRDRLD43YGE2WVDDTGW3FQ, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.898681176 (551h25m28.769764212s) [noise=true , address=145.79.227.18:4007 , piece_id=Y52XM3XDSZFKHR6C7CYQ7VIUQGMWNB3JQMGPRQUVKENFJ7RRSQYQ, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.899027356 (551h25m28.770110391s) [noise=true , address=145.79.227.21:4473 , piece_id=44NH6BNBDRVJMYAJDGHIIWRPHQZGHCA3SIUGQ4IHTAYSMW3U2ESA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.899416557 (551h25m28.770499602s) [noise=true , address=145.79.227.5:4223 , piece_id=T6JVT2OXKEBVYP4ICGLCPK3CLN5H25TFTRXC22E7OBD7NLA3HDLA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.899745403 (551h25m28.770828448s) [noise=true , address=45.91.116.5:4298 , piece_id=F6UV3BDNWFAGWBBJLNSBQETPARFEGKB4Z6POFKVBV6X5DNIFOODA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.900154752 (551h25m28.771237797s) [noise=true , address=145.79.227.44:4194 , piece_id=77E4L3WGPLKI2HUTR6UAOHYTVUMBBGMGVALT6R42L7PLMVIT7JJQ, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.900476626 (551h25m28.771559661s) [noise=true , address=145.79.227.15:4874 , piece_id=3CYMIHKUTOXHXJBVJKUUJC6HKTQA4AD5HCET3UJNDMFHGXGXIOOA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.902303083 (551h25m28.773386118s) [noise=true , address=145.79.227.36:4519 , piece_id=I7VEEU6WTZG4YRMJIKUQTWLPFBECYVYTV4Z43NP3U2QGHPGB3EEA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.903509890 (551h25m28.774592926s) [noise=true , address=77.90.31.13:4626 , piece_id=CG7G54I3IGFS32ZAY2WJHDRCYAGLFVO6TOVKX7R4CUJ6UO7V3G4Q, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.905006530 (551h25m28.776089565s) [noise=true , address=77.90.31.16:4735 , piece_id=Z2JJ7IIWX3RQLXM3HABIULPLEQSMDKVXC6F7YJCW3HCVEBRRYM6Q, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.906620755 (551h25m28.77770379s) [noise=true , address=145.79.227.52:4445 , piece_id=ZKUYH45CZIQNCWP5Q3JMEJTOA5TJSTZL7TTWOBGF5QOP245ZNHHQ, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.908401716 (551h25m28.779484761s) [noise=true , address=145.79.227.2:4113 , piece_id=PIBB5A34I4C3T5RX4DPHMSFBGJGZDGWOP7XBELPTOYA45DRVZZCA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.909977536 (551h25m28.781060571s) [noise=true , address=145.79.227.32:4835 , piece_id=FVGQX4522G64FUDBCKN6LXSTECVOXZONMO3WPUQGWYUMKKLHAL7Q, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.912006570 (551h25m28.783089625s) [noise=true , address=145.79.227.10:4401 , piece_id=A4UVXDWW66OY2DIFDGN47XYGK255NTLGR2U36NUDF32AOJD2LFKA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:30.616536389 (551h25m30.487619424s) [upload=483478]: Next returned nil segment
Dec 28 14:54:30.616555134 (551h25m30.487638179s) [upload=483478]: Waiting for error group managing segments...
Dec 28 14:54:30.902690564 (551h25m30.773773619s) [noise=true , address=145.79.227.5:4223 , piece_id=T6JVT2OXKEBVYP4ICGLCPK3CLN5H25TFTRXC22E7OBD7NLA3HDLA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.908186579 (551h25m30.779269624s) [noise=true , address=145.79.227.2:4113 , piece_id=PIBB5A34I4C3T5RX4DPHMSFBGJGZDGWOP7XBELPTOYA45DRVZZCA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.908476522 (551h25m30.779559557s) [noise=true , address=145.79.227.21:4473 , piece_id=44NH6BNBDRVJMYAJDGHIIWRPHQZGHCA3SIUGQ4IHTAYSMW3U2ESA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.910250058 (551h25m30.781333113s) [noise=true , address=145.79.227.18:4007 , piece_id=Y52XM3XDSZFKHR6C7CYQ7VIUQGMWNB3JQMGPRQUVKENFJ7RRSQYQ, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.911106399 (551h25m30.782189444s) [noise=true , address=145.79.227.50:4373 , piece_id=F5HNDNMLFF4GW3EVG2DR2V2I7D6OZN7NA7B6WZMKWGTSZ5FPKREA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.914846541 (551h25m30.785929587s) [noise=true , address=145.79.227.10:4401 , piece_id=A4UVXDWW66OY2DIFDGN47XYGK255NTLGR2U36NUDF32AOJD2LFKA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.919475566 (551h25m30.790558621s) [noise=true , address=145.79.227.52:4445 , piece_id=ZKUYH45CZIQNCWP5Q3JMEJTOA5TJSTZL7TTWOBGF5QOP245ZNHHQ, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.921006410 (551h25m30.792089455s) [noise=true , address=145.79.227.36:4519 , piece_id=I7VEEU6WTZG4YRMJIKUQTWLPFBECYVYTV4Z43NP3U2QGHPGB3EEA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.921040545 (551h25m30.7921236s) [noise=true , address=145.79.227.15:4874 , piece_id=3CYMIHKUTOXHXJBVJKUUJC6HKTQA4AD5HCET3UJNDMFHGXGXIOOA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.941624211 (551h25m30.812707256s) [noise=true , address=145.79.227.32:4835 , piece_id=FVGQX4522G64FUDBCKN6LXSTECVOXZONMO3WPUQGWYUMKKLHAL7Q, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:31.182363167 (551h25m31.053446202s) [noise=true , address=77.90.31.16:4735 , piece_id=Z2JJ7IIWX3RQLXM3HABIULPLEQSMDKVXC6F7YJCW3HCVEBRRYM6Q, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:31.261978689 (551h25m31.133061734s) [noise=true , address=77.90.31.13:4626 , piece_id=CG7G54I3IGFS32ZAY2WJHDRCYAGLFVO6TOVKX7R4CUJ6UO7V3G4Q, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:31.276375082 (551h25m31.147458117s) [noise=true , address=45.91.116.7:4010 , piece_id=P3XVLM2MNM224KOHB65XZBQNGQNOVJ37RY2FR4JSPRZPOFZM5W4Q, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:31.276394178 (551h25m31.147477223s) [seg_pos={0 0}, upload=483478]: Segment reached optimal threshold of 13 pieces.
Dec 28 14:54:31.276572508 (551h25m31.147655543s) [noise=true , address=45.91.116.6:4973 , piece_id=ZCNSUED6MUMN4OCGGVXD3RVV6OI3D3DRDRLD43YGE2WVDDTGW3FQ, seg_pos={0 0}, upload=483478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF
Dec 28 14:54:31.276578930 (551h25m31.147661975s) [noise=true , address=45.91.116.6:4973 , piece_id=ZCNSUED6MUMN4OCGGVXD3RVV6OI3D3DRDRLD43YGE2WVDDTGW3FQ, seg_pos={0 0}, upload=483478]: Long tail cancellation due to optimal threshold
Dec 28 14:54:31.276703968 (551h25m31.147787003s) [noise=true , address=45.91.116.5:4298 , piece_id=F6UV3BDNWFAGWBBJLNSBQETPARFEGKB4Z6POFKVBV6X5DNIFOODA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:34120->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:34120->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34120->45.91.116.5:4298: use of closed network connection
Dec 28 14:54:31.276709198 (551h25m31.147792233s) [noise=true , address=45.91.116.5:4298 , piece_id=F6UV3BDNWFAGWBBJLNSBQETPARFEGKB4Z6POFKVBV6X5DNIFOODA, seg_pos={0 0}, upload=483478]: Long tail cancellation due to optimal threshold
Dec 28 14:54:31.277040298 (551h25m31.148123343s) [noise=true , address=145.79.227.44:4194 , piece_id=77E4L3WGPLKI2HUTR6UAOHYTVUMBBGMGVALT6R42L7PLMVIT7JJQ, seg_pos={0 0}, upload=483478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF
Dec 28 14:54:31.277060938 (551h25m31.148143983s) [noise=true , address=145.79.227.44:4194 , piece_id=77E4L3WGPLKI2HUTR6UAOHYTVUMBBGMGVALT6R42L7PLMVIT7JJQ, seg_pos={0 0}, upload=483478]: Long tail cancellation due to optimal threshold
Dec 28 14:54:31.277105372 (551h25m31.148188417s) [seg_pos={0 0}, upload=483478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 14:54:31.277258323 (551h25m31.148341378s) [upload=483478]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 14:54:31.277265838 (551h25m31.148348903s) [upload=483478]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 14:54:31.575778626 (551h25m31.446861691s) [upload=483478]: Done uploading segments...
Dec 28 14:54:31.575803704 (551h25m31.446886749s) [upload=483478]: Upload finished. err: <nil>
Duration: 9.173103637s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
9.183801319s trace:
https://jaeger.dev.storj.io/trace/35a5d4335e2cde85
FAIL (2025-12-28 14:55:33 UTC)
orbiter warning: context deadline exceeded
Duration: 9.183830975s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.X....................X...........X.........................] duration:
5.76459559s trace:
https://jaeger.dev.storj.io/trace/70ba9ccd98ccb34e
FAIL (2025-12-28 14:55:20 UTC)
orbiter warning: context deadline exceeded
Duration: 5.764647269s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
49 out of 60 is failed (threshold=6) [X.X.XXXXXXXXXXXXXXXXXXXXXXX.X.XX.XXXXXXXX.X..XXXXX.X.X.XXXXX] duration:
5.002188291s trace:
https://jaeger.dev.storj.io/trace/131df032dcbcf2b
FAIL (2025-12-28 14:55:08 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.002215763s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [..X....................X...........X........................] duration:
15.795569398s trace:
https://jaeger.dev.storj.io/trace/70ba9ccd98ccb34e
FAIL (2025-12-28 14:55:06 UTC)
orbiter warning: context deadline exceeded
Duration: 15.79560681s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
12.980800329s trace:
https://jaeger.dev.storj.io/trace/35a5d4335e2cde85
FAIL (2025-12-28 14:55:03 UTC)
orbiter warning: context deadline exceeded
Duration: 12.980846778s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
20 out of 60 is failed (threshold=3) [...XX.XX.X.X..XXXXX....X...X..X..X...X...X.X.........X.....X] duration:
30.003626399s trace:
https://jaeger.dev.storj.io/trace/400f8ae82c545e27
FAIL (2025-12-28 14:54:57 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: P3XVLM2MNM224KOHB65XZBQNGQNOVJ37RY2FR4JSPRZPOFZM5W4Q) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: CG7G54I3IGFS32ZAY2WJHDRCYAGLFVO6TOVKX7R4CUJ6UO7V3G4Q) context deadline exceeded
======LOGS======
28 14:54:28.651637995 (551h25m28.52272104s) [upload=483478]: Starting upload
Dec 28 14:54:28.651775788 (551h25m28.522858823s) [upload=483478]: Done starting upload
Dec 28 14:54:28.651857663 (551h25m28.522940698s) [upload=483478]: Uploading segments...
Dec 28 14:54:28.651966430 (551h25m28.523049475s) [upload=483478]: Got next segment. Inline: false
Dec 28 14:54:28.652001437 (551h25m28.523084472s) [upload=483478]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 14:54:28.652006597 (551h25m28.523089632s) [upload=483478]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 14:54:28.897465992 (551h25m28.768549038s) [seg_pos={0 0}, upload=483478]: Begin upload segment...
Dec 28 14:54:28.897524775 (551h25m28.76860782s) [seg_pos={0 0}, upload=483478]: Done begin upload segment.
Dec 28 14:54:28.897584187 (551h25m28.768667232s) [noise=true , address=145.79.227.50:4373 , piece_id=F5HNDNMLFF4GW3EVG2DR2V2I7D6OZN7NA7B6WZMKWGTSZ5FPKREA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.897595700 (551h25m28.768678735s) [noise=true , address=45.91.116.7:4010 , piece_id=P3XVLM2MNM224KOHB65XZBQNGQNOVJ37RY2FR4JSPRZPOFZM5W4Q, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.898192205 (551h25m28.76927524s) [noise=true , address=45.91.116.6:4973 , piece_id=ZCNSUED6MUMN4OCGGVXD3RVV6OI3D3DRDRLD43YGE2WVDDTGW3FQ, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.898681176 (551h25m28.769764212s) [noise=true , address=145.79.227.18:4007 , piece_id=Y52XM3XDSZFKHR6C7CYQ7VIUQGMWNB3JQMGPRQUVKENFJ7RRSQYQ, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.899027356 (551h25m28.770110391s) [noise=true , address=145.79.227.21:4473 , piece_id=44NH6BNBDRVJMYAJDGHIIWRPHQZGHCA3SIUGQ4IHTAYSMW3U2ESA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.899416557 (551h25m28.770499602s) [noise=true , address=145.79.227.5:4223 , piece_id=T6JVT2OXKEBVYP4ICGLCPK3CLN5H25TFTRXC22E7OBD7NLA3HDLA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.899745403 (551h25m28.770828448s) [noise=true , address=45.91.116.5:4298 , piece_id=F6UV3BDNWFAGWBBJLNSBQETPARFEGKB4Z6POFKVBV6X5DNIFOODA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.900154752 (551h25m28.771237797s) [noise=true , address=145.79.227.44:4194 , piece_id=77E4L3WGPLKI2HUTR6UAOHYTVUMBBGMGVALT6R42L7PLMVIT7JJQ, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.900476626 (551h25m28.771559661s) [noise=true , address=145.79.227.15:4874 , piece_id=3CYMIHKUTOXHXJBVJKUUJC6HKTQA4AD5HCET3UJNDMFHGXGXIOOA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.902303083 (551h25m28.773386118s) [noise=true , address=145.79.227.36:4519 , piece_id=I7VEEU6WTZG4YRMJIKUQTWLPFBECYVYTV4Z43NP3U2QGHPGB3EEA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.903509890 (551h25m28.774592926s) [noise=true , address=77.90.31.13:4626 , piece_id=CG7G54I3IGFS32ZAY2WJHDRCYAGLFVO6TOVKX7R4CUJ6UO7V3G4Q, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.905006530 (551h25m28.776089565s) [noise=true , address=77.90.31.16:4735 , piece_id=Z2JJ7IIWX3RQLXM3HABIULPLEQSMDKVXC6F7YJCW3HCVEBRRYM6Q, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.906620755 (551h25m28.77770379s) [noise=true , address=145.79.227.52:4445 , piece_id=ZKUYH45CZIQNCWP5Q3JMEJTOA5TJSTZL7TTWOBGF5QOP245ZNHHQ, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.908401716 (551h25m28.779484761s) [noise=true , address=145.79.227.2:4113 , piece_id=PIBB5A34I4C3T5RX4DPHMSFBGJGZDGWOP7XBELPTOYA45DRVZZCA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.909977536 (551h25m28.781060571s) [noise=true , address=145.79.227.32:4835 , piece_id=FVGQX4522G64FUDBCKN6LXSTECVOXZONMO3WPUQGWYUMKKLHAL7Q, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:28.912006570 (551h25m28.783089625s) [noise=true , address=145.79.227.10:4401 , piece_id=A4UVXDWW66OY2DIFDGN47XYGK255NTLGR2U36NUDF32AOJD2LFKA, seg_pos={0 0}, upload=483478]: Uploading piece...
Dec 28 14:54:30.616536389 (551h25m30.487619424s) [upload=483478]: Next returned nil segment
Dec 28 14:54:30.616555134 (551h25m30.487638179s) [upload=483478]: Waiting for error group managing segments...
Dec 28 14:54:30.902690564 (551h25m30.773773619s) [noise=true , address=145.79.227.5:4223 , piece_id=T6JVT2OXKEBVYP4ICGLCPK3CLN5H25TFTRXC22E7OBD7NLA3HDLA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.908186579 (551h25m30.779269624s) [noise=true , address=145.79.227.2:4113 , piece_id=PIBB5A34I4C3T5RX4DPHMSFBGJGZDGWOP7XBELPTOYA45DRVZZCA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.908476522 (551h25m30.779559557s) [noise=true , address=145.79.227.21:4473 , piece_id=44NH6BNBDRVJMYAJDGHIIWRPHQZGHCA3SIUGQ4IHTAYSMW3U2ESA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.910250058 (551h25m30.781333113s) [noise=true , address=145.79.227.18:4007 , piece_id=Y52XM3XDSZFKHR6C7CYQ7VIUQGMWNB3JQMGPRQUVKENFJ7RRSQYQ, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.911106399 (551h25m30.782189444s) [noise=true , address=145.79.227.50:4373 , piece_id=F5HNDNMLFF4GW3EVG2DR2V2I7D6OZN7NA7B6WZMKWGTSZ5FPKREA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.914846541 (551h25m30.785929587s) [noise=true , address=145.79.227.10:4401 , piece_id=A4UVXDWW66OY2DIFDGN47XYGK255NTLGR2U36NUDF32AOJD2LFKA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.919475566 (551h25m30.790558621s) [noise=true , address=145.79.227.52:4445 , piece_id=ZKUYH45CZIQNCWP5Q3JMEJTOA5TJSTZL7TTWOBGF5QOP245ZNHHQ, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.921006410 (551h25m30.792089455s) [noise=true , address=145.79.227.36:4519 , piece_id=I7VEEU6WTZG4YRMJIKUQTWLPFBECYVYTV4Z43NP3U2QGHPGB3EEA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.921040545 (551h25m30.7921236s) [noise=true , address=145.79.227.15:4874 , piece_id=3CYMIHKUTOXHXJBVJKUUJC6HKTQA4AD5HCET3UJNDMFHGXGXIOOA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:30.941624211 (551h25m30.812707256s) [noise=true , address=145.79.227.32:4835 , piece_id=FVGQX4522G64FUDBCKN6LXSTECVOXZONMO3WPUQGWYUMKKLHAL7Q, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:31.182363167 (551h25m31.053446202s) [noise=true , address=77.90.31.16:4735 , piece_id=Z2JJ7IIWX3RQLXM3HABIULPLEQSMDKVXC6F7YJCW3HCVEBRRYM6Q, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:31.261978689 (551h25m31.133061734s) [noise=true , address=77.90.31.13:4626 , piece_id=CG7G54I3IGFS32ZAY2WJHDRCYAGLFVO6TOVKX7R4CUJ6UO7V3G4Q, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:31.276375082 (551h25m31.147458117s) [noise=true , address=45.91.116.7:4010 , piece_id=P3XVLM2MNM224KOHB65XZBQNGQNOVJ37RY2FR4JSPRZPOFZM5W4Q, seg_pos={0 0}, upload=483478]: Done uploading piece. err: <nil>
Dec 28 14:54:31.276394178 (551h25m31.147477223s) [seg_pos={0 0}, upload=483478]: Segment reached optimal threshold of 13 pieces.
Dec 28 14:54:31.276572508 (551h25m31.147655543s) [noise=true , address=45.91.116.6:4973 , piece_id=ZCNSUED6MUMN4OCGGVXD3RVV6OI3D3DRDRLD43YGE2WVDDTGW3FQ, seg_pos={0 0}, upload=483478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF
Dec 28 14:54:31.276578930 (551h25m31.147661975s) [noise=true , address=45.91.116.6:4973 , piece_id=ZCNSUED6MUMN4OCGGVXD3RVV6OI3D3DRDRLD43YGE2WVDDTGW3FQ, seg_pos={0 0}, upload=483478]: Long tail cancellation due to optimal threshold
Dec 28 14:54:31.276703968 (551h25m31.147787003s) [noise=true , address=45.91.116.5:4298 , piece_id=F6UV3BDNWFAGWBBJLNSBQETPARFEGKB4Z6POFKVBV6X5DNIFOODA, seg_pos={0 0}, upload=483478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:34120->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:34120->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34120->45.91.116.5:4298: use of closed network connection
Dec 28 14:54:31.276709198 (551h25m31.147792233s) [noise=true , address=45.91.116.5:4298 , piece_id=F6UV3BDNWFAGWBBJLNSBQETPARFEGKB4Z6POFKVBV6X5DNIFOODA, seg_pos={0 0}, upload=483478]: Long tail cancellation due to optimal threshold
Dec 28 14:54:31.277040298 (551h25m31.148123343s) [noise=true , address=145.79.227.44:4194 , piece_id=77E4L3WGPLKI2HUTR6UAOHYTVUMBBGMGVALT6R42L7PLMVIT7JJQ, seg_pos={0 0}, upload=483478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF
Dec 28 14:54:31.277060938 (551h25m31.148143983s) [noise=true , address=145.79.227.44:4194 , piece_id=77E4L3WGPLKI2HUTR6UAOHYTVUMBBGMGVALT6R42L7PLMVIT7JJQ, seg_pos={0 0}, upload=483478]: Long tail cancellation due to optimal threshold
Dec 28 14:54:31.277105372 (551h25m31.148188417s) [seg_pos={0 0}, upload=483478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 14:54:31.277258323 (551h25m31.148341378s) [upload=483478]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 14:54:31.277265838 (551h25m31.148348903s) [upload=483478]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 14:54:31.575778626 (551h25m31.446861691s) [upload=483478]: Done uploading segments...
Dec 28 14:54:31.575803704 (551h25m31.446886749s) [upload=483478]: Upload finished. err: <nil>
Duration: 30.003649994s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [.....X...X............X...............X......X....X.........] duration:
47.808983237s trace:
https://jaeger.dev.storj.io/trace/1440a926e26dd799
FAIL (2025-12-28 14:54:54 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
noise=true , address=145.79.227.44:4212 , piece_id=V73IIIU7CCTULBPEXD2YWSFUUGEP2ETVL7LQGIWWEXICZ5SK4BQA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.239195729 (551h16m25.110278774s) [noise=true , address=145.79.227.34:4931 , piece_id=IBNDRK5G5MGWQKRTJ3CS7LI7F3TFUJDHZDG6HJZ44XOQAEWBVEBA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.239549302 (551h16m25.110632347s) [noise=true , address=64.49.13.140:4515 , piece_id=42YNEU3HOVPRPUVOCPKYSQUBTDLBQELQCWNIYJBDXD2RV2NRDJDQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.239966557 (551h16m25.111049602s) [noise=true , address=77.90.31.18:4816 , piece_id=J3DJPEVEESV6EA5KJU6TN7MO4ATQZDBQOSJSH5S5KSZRYGHHZRSQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.240145648 (551h16m25.111228683s) [noise=true , address=77.90.31.14:4688 , piece_id=LNETNIH6PNQ2OR2NQKBWWWOJEFYXIGYHHH34WW7DHTKKYQWJ6DLA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.240933478 (551h16m25.112016523s) [noise=true , address=77.90.31.19:4867 , piece_id=NWQD7PXUGDHUKRF4J3E6EUORCKUM3TDAD2EJH7F3CE55G4EA7SJQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.241389286 (551h16m25.112472331s) [noise=true , address=145.79.227.40:4075 , piece_id=XKJZ2GSNELWKMTMZA5EZ2YTM62XREZQDFI7MDDKAPE3YXRY4KWNA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.241451415 (551h16m25.11253445s) [noise=true , address=77.90.31.21:4889 , piece_id=WSIKD7GCMKHWEFYC7UZ3YWGQHQDIZVW2YJY4BD5ENMCYBJZJMXEQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.241843531 (551h16m25.112926566s) [noise=true , address=145.79.227.46:4608 , piece_id=T5G4T4OU62UPXPIACXJQEZ36O2N63HL4L4HRMW3EN6B4PIE3M7YA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.241911240 (551h16m25.112994285s) [noise=true , address=64.49.13.169:4711 , piece_id=YHY2V4EHCMQEDR6OTZE2POM45Y7FYVNOSEGSUOM5IYWCDWDLRIKQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.242196533 (551h16m25.113279578s) [noise=true , address=193.149.164.143:4697 , piece_id=6SHPKJO6AZLKYPIQWJWNI4YSAMLAVU7BGDGQLQC4I6AGCRWT52DA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.242532343 (551h16m25.113615378s) [noise=true , address=145.79.227.29:4947 , piece_id=FSJZQVSKN76O5RLIVQDPKAN2PDV4VLRCMBRA5EYJGFOETOZ6D7AQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.242582478 (551h16m25.113665613s) [noise=true , address=145.79.227.18:4010 , piece_id=GVY67LYRK3UZRX6OLLWC5JCJEGV3RCIS6DATIDMQCBJSL7DCAE4Q, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.242986888 (551h16m25.114069923s) [noise=true , address=64.49.13.27:4966 , piece_id=N3R7JO5PL47V7ZUXNLXFOHSI2MEJ2SVQNHE6M2DWDYF3D4L34ZCA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.243048475 (551h16m25.11413151s) [noise=true , address=77.90.31.11:4585 , piece_id=5MKOQOD3O7TXKXODJWTVZI5EAKBEO7JE3KKXD63F6SO3EBBSVD6A, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.243336464 (551h16m25.114419499s) [noise=true , address=77.90.31.17:4801 , piece_id=OWQPIGFIL5SFLUI77IYA3LKVUFPSBL6JSJO6XWZLQFV7GXHF3QOA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:26.537548687 (551h16m26.408631722s) [upload=483348]: Next returned nil segment
Dec 28 14:45:26.537559538 (551h16m26.408642573s) [upload=483348]: Waiting for error group managing segments...
Dec 28 14:45:26.899924119 (551h16m26.771007164s) [noise=true , address=64.49.13.140:4515 , piece_id=42YNEU3HOVPRPUVOCPKYSQUBTDLBQELQCWNIYJBDXD2RV2NRDJDQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:26.977957828 (551h16m26.849040873s) [noise=true , address=64.49.13.27:4966 , piece_id=N3R7JO5PL47V7ZUXNLXFOHSI2MEJ2SVQNHE6M2DWDYF3D4L34ZCA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.120995816 (551h16m26.992078862s) [noise=true , address=145.79.227.46:4608 , piece_id=T5G4T4OU62UPXPIACXJQEZ36O2N63HL4L4HRMW3EN6B4PIE3M7YA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.151904071 (551h16m27.022987116s) [noise=true , address=145.79.227.34:4931 , piece_id=IBNDRK5G5MGWQKRTJ3CS7LI7F3TFUJDHZDG6HJZ44XOQAEWBVEBA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.165435146 (551h16m27.036518181s) [noise=true , address=145.79.227.29:4947 , piece_id=FSJZQVSKN76O5RLIVQDPKAN2PDV4VLRCMBRA5EYJGFOETOZ6D7AQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.175130708 (551h16m27.046213754s) [noise=true , address=145.79.227.18:4010 , piece_id=GVY67LYRK3UZRX6OLLWC5JCJEGV3RCIS6DATIDMQCBJSL7DCAE4Q, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.352651587 (551h16m27.223734632s) [noise=true , address=77.90.31.19:4867 , piece_id=NWQD7PXUGDHUKRF4J3E6EUORCKUM3TDAD2EJH7F3CE55G4EA7SJQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.375485928 (551h16m27.246568973s) [noise=true , address=77.90.31.11:4585 , piece_id=5MKOQOD3O7TXKXODJWTVZI5EAKBEO7JE3KKXD63F6SO3EBBSVD6A, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.428947778 (551h16m27.300030823s) [noise=true , address=77.90.31.18:4816 , piece_id=J3DJPEVEESV6EA5KJU6TN7MO4ATQZDBQOSJSH5S5KSZRYGHHZRSQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.567872934 (551h16m27.438955969s) [noise=true , address=77.90.31.17:4801 , piece_id=OWQPIGFIL5SFLUI77IYA3LKVUFPSBL6JSJO6XWZLQFV7GXHF3QOA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.983670843 (551h16m27.854753878s) [noise=true , address=77.90.31.13:4650 , piece_id=NHTWFNKCRHELW7BJ2NR4GGHY4LU743RVLCCR236AO4MQUZD7KMEQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:28.019933813 (551h16m27.891016848s) [noise=true , address=145.79.227.30:4794 , piece_id=OZXLKN4V62NRD3WCMF65C7U7OPM2FPBOIMNNE53LN4MBSDDL65NQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:28.083140990 (551h16m27.954224025s) [noise=true , address=145.79.227.44:4212 , piece_id=V73IIIU7CCTULBPEXD2YWSFUUGEP2ETVL7LQGIWWEXICZ5SK4BQA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:28.129315084 (551h16m28.000398119s) [noise=true , address=145.79.227.40:4075 , piece_id=XKJZ2GSNELWKMTMZA5EZ2YTM62XREZQDFI7MDDKAPE3YXRY4KWNA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:28.129340211 (551h16m28.000423246s) [seg_pos={0 0}, upload=483348]: Segment reached optimal threshold of 14 pieces.
Dec 28 14:45:28.129532938 (551h16m28.000615973s) [noise=true , address=77.90.31.21:4889 , piece_id=WSIKD7GCMKHWEFYC7UZ3YWGQHQDIZVW2YJY4BD5ENMCYBJZJMXEQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wcX8iEhi5LrftcXUQE3Lrm4igNwhDtvT3RNBERTGsKyQ6KLj6): protocol: expected piece hash; context canceled; EOF
Dec 28 14:45:28.129539380 (551h16m28.000622425s) [noise=true , address=77.90.31.21:4889 , piece_id=WSIKD7GCMKHWEFYC7UZ3YWGQHQDIZVW2YJY4BD5ENMCYBJZJMXEQ, seg_pos={0 0}, upload=483348]: Long tail cancellation due to optimal threshold
Dec 28 14:45:28.129666382 (551h16m28.000749418s) [noise=true , address=64.49.13.169:4711 , piece_id=YHY2V4EHCMQEDR6OTZE2POM45Y7FYVNOSEGSUOM5IYWCDWDLRIKQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:47258->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:47258->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47258->64.49.13.169:4711: use of closed network connection
Dec 28 14:45:28.129671853 (551h16m28.000754898s) [noise=true , address=64.49.13.169:4711 , piece_id=YHY2V4EHCMQEDR6OTZE2POM45Y7FYVNOSEGSUOM5IYWCDWDLRIKQ, seg_pos={0 0}, upload=483348]: Long tail cancellation due to optimal threshold
Dec 28 14:45:28.129713071 (551h16m28.000796116s) [noise=true , address=77.90.31.14:4688 , piece_id=LNETNIH6PNQ2OR2NQKBWWWOJEFYXIGYHHH34WW7DHTKKYQWJ6DLA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz): protocol: expected piece hash; context canceled; EOF
Dec 28 14:45:28.129770090 (551h16m28.000853125s) [noise=true , address=77.90.31.14:4688 , piece_id=LNETNIH6PNQ2OR2NQKBWWWOJEFYXIGYHHH34WW7DHTKKYQWJ6DLA, seg_pos={0 0}, upload=483348]: Long tail cancellation due to optimal threshold
Dec 28 14:45:28.129862666 (551h16m28.000945711s) [noise=true , address=193.149.164.143:4697 , piece_id=6SHPKJO6AZLKYPIQWJWNI4YSAMLAVU7BGDGQLQC4I6AGCRWT52DA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ): protocol: expected piece hash; context canceled; EOF
Dec 28 14:45:28.129870651 (551h16m28.000953686s) [noise=true , address=193.149.164.143:4697 , piece_id=6SHPKJO6AZLKYPIQWJWNI4YSAMLAVU7BGDGQLQC4I6AGCRWT52DA, seg_pos={0 0}, upload=483348]: Long tail cancellation due to optimal threshold
Dec 28 14:45:28.129970341 (551h16m28.001053376s) [seg_pos={0 0}, upload=483348]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 14:45:28.130053620 (551h16m28.001136665s) [upload=483348]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 14:45:28.130060553 (551h16m28.001143588s) [upload=483348]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 14:45:28.417819022 (551h16m28.288902057s) [upload=483348]: Done uploading segments...
Dec 28 14:45:28.417844872 (551h16m28.288927927s) [upload=483348]: Upload finished. err: <nil>
Duration: 47.809013674s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
14.323763435s trace:
https://jaeger.dev.storj.io/trace/35a5d4335e2cde85
FAIL (2025-12-28 14:54:46 UTC)
orbiter warning: context deadline exceeded
Duration: 14.323789846s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
49 out of 60 is failed (threshold=6) [XX.X.XXXXXXXXXXXXXXXXXXXXXXX.X.XX.XXXXXXXX.X..XXXXX.X.X.XXXX] duration:
5.001522073s trace:
https://jaeger.dev.storj.io/trace/4b566b1ab57d450f
FAIL (2025-12-28 14:54:42 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001548091s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [...X....................X...........X.......................] duration:
15.212283084s trace:
https://jaeger.dev.storj.io/trace/70ba9ccd98ccb34e
FAIL (2025-12-28 14:54:07 UTC)
orbiter warning: context deadline exceeded
Duration: 15.212318111s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
18.982387585s trace:
https://jaeger.dev.storj.io/trace/35a5d4335e2cde85
FAIL (2025-12-28 14:53:56 UTC)
orbiter warning: context deadline exceeded
Duration: 18.982414726s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
49 out of 60 is failed (threshold=6) [XXX.X.XXXXXXXXXXXXXXXXXXXXXXX.X.XX.XXXXXXXX.X..XXXXX.X.X.XXX] duration:
5.001177619s trace:
https://jaeger.dev.storj.io/trace/5ce165c9bc2b7b99
FAIL (2025-12-28 14:53:49 UTC)
orbiter warning: context deadline exceeded
Duration: 5.00119925s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [......X...X............X...............X......X....X........] duration:
38.185291722s trace:
https://jaeger.dev.storj.io/trace/1440a926e26dd799
FAIL (2025-12-28 14:53:33 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
noise=true , address=145.79.227.44:4212 , piece_id=V73IIIU7CCTULBPEXD2YWSFUUGEP2ETVL7LQGIWWEXICZ5SK4BQA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.239195729 (551h16m25.110278774s) [noise=true , address=145.79.227.34:4931 , piece_id=IBNDRK5G5MGWQKRTJ3CS7LI7F3TFUJDHZDG6HJZ44XOQAEWBVEBA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.239549302 (551h16m25.110632347s) [noise=true , address=64.49.13.140:4515 , piece_id=42YNEU3HOVPRPUVOCPKYSQUBTDLBQELQCWNIYJBDXD2RV2NRDJDQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.239966557 (551h16m25.111049602s) [noise=true , address=77.90.31.18:4816 , piece_id=J3DJPEVEESV6EA5KJU6TN7MO4ATQZDBQOSJSH5S5KSZRYGHHZRSQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.240145648 (551h16m25.111228683s) [noise=true , address=77.90.31.14:4688 , piece_id=LNETNIH6PNQ2OR2NQKBWWWOJEFYXIGYHHH34WW7DHTKKYQWJ6DLA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.240933478 (551h16m25.112016523s) [noise=true , address=77.90.31.19:4867 , piece_id=NWQD7PXUGDHUKRF4J3E6EUORCKUM3TDAD2EJH7F3CE55G4EA7SJQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.241389286 (551h16m25.112472331s) [noise=true , address=145.79.227.40:4075 , piece_id=XKJZ2GSNELWKMTMZA5EZ2YTM62XREZQDFI7MDDKAPE3YXRY4KWNA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.241451415 (551h16m25.11253445s) [noise=true , address=77.90.31.21:4889 , piece_id=WSIKD7GCMKHWEFYC7UZ3YWGQHQDIZVW2YJY4BD5ENMCYBJZJMXEQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.241843531 (551h16m25.112926566s) [noise=true , address=145.79.227.46:4608 , piece_id=T5G4T4OU62UPXPIACXJQEZ36O2N63HL4L4HRMW3EN6B4PIE3M7YA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.241911240 (551h16m25.112994285s) [noise=true , address=64.49.13.169:4711 , piece_id=YHY2V4EHCMQEDR6OTZE2POM45Y7FYVNOSEGSUOM5IYWCDWDLRIKQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.242196533 (551h16m25.113279578s) [noise=true , address=193.149.164.143:4697 , piece_id=6SHPKJO6AZLKYPIQWJWNI4YSAMLAVU7BGDGQLQC4I6AGCRWT52DA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.242532343 (551h16m25.113615378s) [noise=true , address=145.79.227.29:4947 , piece_id=FSJZQVSKN76O5RLIVQDPKAN2PDV4VLRCMBRA5EYJGFOETOZ6D7AQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.242582478 (551h16m25.113665613s) [noise=true , address=145.79.227.18:4010 , piece_id=GVY67LYRK3UZRX6OLLWC5JCJEGV3RCIS6DATIDMQCBJSL7DCAE4Q, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.242986888 (551h16m25.114069923s) [noise=true , address=64.49.13.27:4966 , piece_id=N3R7JO5PL47V7ZUXNLXFOHSI2MEJ2SVQNHE6M2DWDYF3D4L34ZCA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.243048475 (551h16m25.11413151s) [noise=true , address=77.90.31.11:4585 , piece_id=5MKOQOD3O7TXKXODJWTVZI5EAKBEO7JE3KKXD63F6SO3EBBSVD6A, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.243336464 (551h16m25.114419499s) [noise=true , address=77.90.31.17:4801 , piece_id=OWQPIGFIL5SFLUI77IYA3LKVUFPSBL6JSJO6XWZLQFV7GXHF3QOA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:26.537548687 (551h16m26.408631722s) [upload=483348]: Next returned nil segment
Dec 28 14:45:26.537559538 (551h16m26.408642573s) [upload=483348]: Waiting for error group managing segments...
Dec 28 14:45:26.899924119 (551h16m26.771007164s) [noise=true , address=64.49.13.140:4515 , piece_id=42YNEU3HOVPRPUVOCPKYSQUBTDLBQELQCWNIYJBDXD2RV2NRDJDQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:26.977957828 (551h16m26.849040873s) [noise=true , address=64.49.13.27:4966 , piece_id=N3R7JO5PL47V7ZUXNLXFOHSI2MEJ2SVQNHE6M2DWDYF3D4L34ZCA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.120995816 (551h16m26.992078862s) [noise=true , address=145.79.227.46:4608 , piece_id=T5G4T4OU62UPXPIACXJQEZ36O2N63HL4L4HRMW3EN6B4PIE3M7YA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.151904071 (551h16m27.022987116s) [noise=true , address=145.79.227.34:4931 , piece_id=IBNDRK5G5MGWQKRTJ3CS7LI7F3TFUJDHZDG6HJZ44XOQAEWBVEBA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.165435146 (551h16m27.036518181s) [noise=true , address=145.79.227.29:4947 , piece_id=FSJZQVSKN76O5RLIVQDPKAN2PDV4VLRCMBRA5EYJGFOETOZ6D7AQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.175130708 (551h16m27.046213754s) [noise=true , address=145.79.227.18:4010 , piece_id=GVY67LYRK3UZRX6OLLWC5JCJEGV3RCIS6DATIDMQCBJSL7DCAE4Q, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.352651587 (551h16m27.223734632s) [noise=true , address=77.90.31.19:4867 , piece_id=NWQD7PXUGDHUKRF4J3E6EUORCKUM3TDAD2EJH7F3CE55G4EA7SJQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.375485928 (551h16m27.246568973s) [noise=true , address=77.90.31.11:4585 , piece_id=5MKOQOD3O7TXKXODJWTVZI5EAKBEO7JE3KKXD63F6SO3EBBSVD6A, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.428947778 (551h16m27.300030823s) [noise=true , address=77.90.31.18:4816 , piece_id=J3DJPEVEESV6EA5KJU6TN7MO4ATQZDBQOSJSH5S5KSZRYGHHZRSQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.567872934 (551h16m27.438955969s) [noise=true , address=77.90.31.17:4801 , piece_id=OWQPIGFIL5SFLUI77IYA3LKVUFPSBL6JSJO6XWZLQFV7GXHF3QOA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.983670843 (551h16m27.854753878s) [noise=true , address=77.90.31.13:4650 , piece_id=NHTWFNKCRHELW7BJ2NR4GGHY4LU743RVLCCR236AO4MQUZD7KMEQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:28.019933813 (551h16m27.891016848s) [noise=true , address=145.79.227.30:4794 , piece_id=OZXLKN4V62NRD3WCMF65C7U7OPM2FPBOIMNNE53LN4MBSDDL65NQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:28.083140990 (551h16m27.954224025s) [noise=true , address=145.79.227.44:4212 , piece_id=V73IIIU7CCTULBPEXD2YWSFUUGEP2ETVL7LQGIWWEXICZ5SK4BQA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:28.129315084 (551h16m28.000398119s) [noise=true , address=145.79.227.40:4075 , piece_id=XKJZ2GSNELWKMTMZA5EZ2YTM62XREZQDFI7MDDKAPE3YXRY4KWNA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:28.129340211 (551h16m28.000423246s) [seg_pos={0 0}, upload=483348]: Segment reached optimal threshold of 14 pieces.
Dec 28 14:45:28.129532938 (551h16m28.000615973s) [noise=true , address=77.90.31.21:4889 , piece_id=WSIKD7GCMKHWEFYC7UZ3YWGQHQDIZVW2YJY4BD5ENMCYBJZJMXEQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wcX8iEhi5LrftcXUQE3Lrm4igNwhDtvT3RNBERTGsKyQ6KLj6): protocol: expected piece hash; context canceled; EOF
Dec 28 14:45:28.129539380 (551h16m28.000622425s) [noise=true , address=77.90.31.21:4889 , piece_id=WSIKD7GCMKHWEFYC7UZ3YWGQHQDIZVW2YJY4BD5ENMCYBJZJMXEQ, seg_pos={0 0}, upload=483348]: Long tail cancellation due to optimal threshold
Dec 28 14:45:28.129666382 (551h16m28.000749418s) [noise=true , address=64.49.13.169:4711 , piece_id=YHY2V4EHCMQEDR6OTZE2POM45Y7FYVNOSEGSUOM5IYWCDWDLRIKQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:47258->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:47258->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47258->64.49.13.169:4711: use of closed network connection
Dec 28 14:45:28.129671853 (551h16m28.000754898s) [noise=true , address=64.49.13.169:4711 , piece_id=YHY2V4EHCMQEDR6OTZE2POM45Y7FYVNOSEGSUOM5IYWCDWDLRIKQ, seg_pos={0 0}, upload=483348]: Long tail cancellation due to optimal threshold
Dec 28 14:45:28.129713071 (551h16m28.000796116s) [noise=true , address=77.90.31.14:4688 , piece_id=LNETNIH6PNQ2OR2NQKBWWWOJEFYXIGYHHH34WW7DHTKKYQWJ6DLA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz): protocol: expected piece hash; context canceled; EOF
Dec 28 14:45:28.129770090 (551h16m28.000853125s) [noise=true , address=77.90.31.14:4688 , piece_id=LNETNIH6PNQ2OR2NQKBWWWOJEFYXIGYHHH34WW7DHTKKYQWJ6DLA, seg_pos={0 0}, upload=483348]: Long tail cancellation due to optimal threshold
Dec 28 14:45:28.129862666 (551h16m28.000945711s) [noise=true , address=193.149.164.143:4697 , piece_id=6SHPKJO6AZLKYPIQWJWNI4YSAMLAVU7BGDGQLQC4I6AGCRWT52DA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ): protocol: expected piece hash; context canceled; EOF
Dec 28 14:45:28.129870651 (551h16m28.000953686s) [noise=true , address=193.149.164.143:4697 , piece_id=6SHPKJO6AZLKYPIQWJWNI4YSAMLAVU7BGDGQLQC4I6AGCRWT52DA, seg_pos={0 0}, upload=483348]: Long tail cancellation due to optimal threshold
Dec 28 14:45:28.129970341 (551h16m28.001053376s) [seg_pos={0 0}, upload=483348]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 14:45:28.130053620 (551h16m28.001136665s) [upload=483348]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 14:45:28.130060553 (551h16m28.001143588s) [upload=483348]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 14:45:28.417819022 (551h16m28.288902057s) [upload=483348]: Done uploading segments...
Dec 28 14:45:28.417844872 (551h16m28.288927927s) [upload=483348]: Upload finished. err: <nil>
Duration: 38.185313595s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [....XX.XX.X.X..XXXXX....X...X..X..X...X...X.X.........X.....] duration:
5.680302289s trace:
https://jaeger.dev.storj.io/trace/550d1e289028a496
FAIL (2025-12-28 14:53:23 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 28 14:49:26.847085842 (551h20m26.718168877s) [upload=483402]: Starting upload
Dec 28 14:49:26.847246297 (551h20m26.718329332s) [upload=483402]: Done starting upload
Dec 28 14:49:26.847480553 (551h20m26.718563588s) [upload=483402]: Uploading segments...
Dec 28 14:49:26.847534546 (551h20m26.718617581s) [upload=483402]: Got next segment. Inline: false
Dec 28 14:49:26.847611382 (551h20m26.718694418s) [upload=483402]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 14:49:26.847625389 (551h20m26.718708424s) [upload=483402]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 14:49:27.094997497 (551h20m26.966080542s) [seg_pos={0 0}, upload=483402]: Begin upload segment...
Dec 28 14:49:27.095052100 (551h20m26.966135145s) [seg_pos={0 0}, upload=483402]: Done begin upload segment.
Dec 28 14:49:27.095090593 (551h20m26.966173638s) [noise=true , address=45.91.116.13:4156 , piece_id=SEVVYACKWMXC6I6QQ54JHPNJBTDE3E2V2J7KFL5WTI7QKHGY4YQQ, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.095108959 (551h20m26.966191994s) [noise=true , address=145.79.227.48:4303 , piece_id=YL7JSA5QHJ3GK4NU344BNUIF5TONSV767UTYEE3YMMF4P7BQ35TA, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.095613940 (551h20m26.966696975s) [noise=true , address=145.79.227.39:4015 , piece_id=TXO5233BTSNH5F4KXE62VOFKV3A5WEFLJBOM72L5ROU6C4M457XQ, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.095909944 (551h20m26.966992979s) [noise=true , address=145.79.227.4:4186 , piece_id=E5RL63ZF2X67454ITCBCP77C3RBY5S5CH4HJQT6EDUAAG4K3MVFQ, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.096198974 (551h20m26.967282009s) [noise=true , address=145.79.227.42:4123 , piece_id=KBOYHLQK2JYAAAD7U3QFQ6HP54KTQPIB4JZF2RV4H7BHLE3WPCZQ, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.096496671 (551h20m26.967579706s) [noise=true , address=45.91.116.6:4976 , piece_id=MJ6RW6NMXBDYGUM2IOBH7HNVFVXSDEJL5UFZKP4FODDP6R6G5OYQ, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.096817342 (551h20m26.967900377s) [noise=true , address=145.79.227.19:4077 , piece_id=VU7DE2EBAGLW7STPZJMH3IPR6FXVOQTXUFWTUH7KSLQIBOVS6CRA, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.097201153 (551h20m26.968284188s) [noise=true , address=145.79.227.6:4259 , piece_id=6OUQWEJO3JCVK4R2CSXHI2USMJMVPWYQSCGYYHUKOA3WOJA2XKCQ, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.097541781 (551h20m26.968624826s) [noise=true , address=145.79.227.37:4566 , piece_id=6RH6NXC5R3U25RAQOHDS42KBJ6QWCS24VRCW6GTKTIIG2FFPHGFQ, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.097932225 (551h20m26.96901526s) [noise=true , address=77.90.31.18:4805 , piece_id=7LO5YQSS2UBZIGEZS7F226PPVMTGLMWKFDDUE2A7WAB3766ILBQA, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.098618392 (551h20m26.969701437s) [noise=true , address=45.91.116.5:4297 , piece_id=75ZWT6KXOSDEE4XG44JHCBU4X5TYYY4FYQ5VHIZSHMGDUBQLRRUQ, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.098971544 (551h20m26.970054579s) [noise=true , address=145.79.227.2:4114 , piece_id=VHSQUG2VEO2ZQZ3DHNLNWYTS4YYHFF35FL7KWWU37GCTAZPLMXUQ, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.099341618 (551h20m26.970424653s) [noise=true , address=45.91.116.8:4048 , piece_id=3PB5EN5FVSYQZ4HUJNXA57A4HRTAQP2GQXECS7MRSFRC7CHXQ7KA, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.099693468 (551h20m26.970776503s) [noise=true , address=145.79.227.25:4619 , piece_id=K56Z6MMBFPBQ5C4ZTZ2JWY6N7FQFEQ6Z7PBDOYL6E5CIFG4MA6LA, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.100159947 (551h20m26.971242992s) [noise=true , address=145.79.227.36:4519 , piece_id=JA3NZNENAPXRYQ65A5RJ4AMOYMIQ75F4KTGH4P6GV3YO4PXXCS4Q, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.100510023 (551h20m26.971593058s) [noise=true , address=145.79.227.50:4373 , piece_id=SGLAGZDBNWO6GTYBAD7257DKCXROXZUDN2RTRFX55AY5UVGTX3RQ, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:28.811420760 (551h20m28.682503805s) [upload=483402]: Next returned nil segment
Dec 28 14:49:28.811513175 (551h20m28.68259621s) [upload=483402]: Waiting for error group managing segments...
Dec 28 14:49:29.101125205 (551h20m28.97220825s) [noise=true , address=145.79.227.25:4619 , piece_id=K56Z6MMBFPBQ5C4ZTZ2JWY6N7FQFEQ6Z7PBDOYL6E5CIFG4MA6LA, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.102240679 (551h20m28.973323724s) [noise=true , address=145.79.227.36:4519 , piece_id=JA3NZNENAPXRYQ65A5RJ4AMOYMIQ75F4KTGH4P6GV3YO4PXXCS4Q, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.103529042 (551h20m28.974612097s) [noise=true , address=145.79.227.19:4077 , piece_id=VU7DE2EBAGLW7STPZJMH3IPR6FXVOQTXUFWTUH7KSLQIBOVS6CRA, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.104590202 (551h20m28.975673247s) [noise=true , address=145.79.227.4:4186 , piece_id=E5RL63ZF2X67454ITCBCP77C3RBY5S5CH4HJQT6EDUAAG4K3MVFQ, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.123887168 (551h20m28.994970203s) [noise=true , address=145.79.227.6:4259 , piece_id=6OUQWEJO3JCVK4R2CSXHI2USMJMVPWYQSCGYYHUKOA3WOJA2XKCQ, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.124475989 (551h20m28.995559054s) [noise=true , address=145.79.227.39:4015 , piece_id=TXO5233BTSNH5F4KXE62VOFKV3A5WEFLJBOM72L5ROU6C4M457XQ, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.124745742 (551h20m28.995828787s) [noise=true , address=145.79.227.37:4566 , piece_id=6RH6NXC5R3U25RAQOHDS42KBJ6QWCS24VRCW6GTKTIIG2FFPHGFQ, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.125303544 (551h20m28.996386579s) [noise=true , address=145.79.227.2:4114 , piece_id=VHSQUG2VEO2ZQZ3DHNLNWYTS4YYHFF35FL7KWWU37GCTAZPLMXUQ, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.128525870 (551h20m28.999608915s) [noise=true , address=145.79.227.50:4373 , piece_id=SGLAGZDBNWO6GTYBAD7257DKCXROXZUDN2RTRFX55AY5UVGTX3RQ, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.130796023 (551h20m29.001879078s) [noise=true , address=145.79.227.42:4123 , piece_id=KBOYHLQK2JYAAAD7U3QFQ6HP54KTQPIB4JZF2RV4H7BHLE3WPCZQ, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.131247102 (551h20m29.002330147s) [noise=true , address=145.79.227.48:4303 , piece_id=YL7JSA5QHJ3GK4NU344BNUIF5TONSV767UTYEE3YMMF4P7BQ35TA, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.313107584 (551h20m29.184190639s) [noise=true , address=45.91.116.6:4976 , piece_id=MJ6RW6NMXBDYGUM2IOBH7HNVFVXSDEJL5UFZKP4FODDP6R6G5OYQ, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.375266024 (551h20m29.246349069s) [noise=true , address=45.91.116.13:4156 , piece_id=SEVVYACKWMXC6I6QQ54JHPNJBTDE3E2V2J7KFL5WTI7QKHGY4YQQ, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.375283747 (551h20m29.246366792s) [seg_pos={0 0}, upload=483402]: Segment reached optimal threshold of 13 pieces.
Dec 28 14:49:29.375478719 (551h20m29.246561754s) [noise=true , address=77.90.31.18:4805 , piece_id=7LO5YQSS2UBZIGEZS7F226PPVMTGLMWKFDDUE2A7WAB3766ILBQA, seg_pos={0 0}, upload=483402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF
Dec 28 14:49:29.375485772 (551h20m29.246568807s) [noise=true , address=77.90.31.18:4805 , piece_id=7LO5YQSS2UBZIGEZS7F226PPVMTGLMWKFDDUE2A7WAB3766ILBQA, seg_pos={0 0}, upload=483402]: Long tail cancellation due to optimal threshold
Dec 28 14:49:29.375489529 (551h20m29.246572564s) [noise=true , address=45.91.116.5:4297 , piece_id=75ZWT6KXOSDEE4XG44JHCBU4X5TYYY4FYQ5VHIZSHMGDUBQLRRUQ, seg_pos={0 0}, upload=483402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF
Dec 28 14:49:29.375498176 (551h20m29.246581231s) [noise=true , address=45.91.116.5:4297 , piece_id=75ZWT6KXOSDEE4XG44JHCBU4X5TYYY4FYQ5VHIZSHMGDUBQLRRUQ, seg_pos={0 0}, upload=483402]: Long tail cancellation due to optimal threshold
Dec 28 14:49:29.375598046 (551h20m29.246681091s) [noise=true , address=45.91.116.8:4048 , piece_id=3PB5EN5FVSYQZ4HUJNXA57A4HRTAQP2GQXECS7MRSFRC7CHXQ7KA, seg_pos={0 0}, upload=483402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF
Dec 28 14:49:29.375603437 (551h20m29.246686472s) [noise=true , address=45.91.116.8:4048 , piece_id=3PB5EN5FVSYQZ4HUJNXA57A4HRTAQP2GQXECS7MRSFRC7CHXQ7KA, seg_pos={0 0}, upload=483402]: Long tail cancellation due to optimal threshold
Dec 28 14:49:29.375617824 (551h20m29.246700859s) [seg_pos={0 0}, upload=483402]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 14:49:29.375687867 (551h20m29.246770912s) [upload=483402]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 14:49:29.375692165 (551h20m29.2467752s) [upload=483402]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 14:49:29.682258328 (551h20m29.553341363s) [upload=483402]: Done uploading segments...
Dec 28 14:49:29.682278987 (551h20m29.553362032s) [upload=483402]: Upload finished. err: <nil>
Duration: 5.680327669s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
49 out of 60 is failed (threshold=6) [XXXX.X.XXXXXXXXXXXXXXXXXXXXXXX.X.XX.XXXXXXXX.X..XXXXX.X.X.XX] duration:
5.001689061s trace:
https://jaeger.dev.storj.io/trace/d524e3f8683c388
FAIL (2025-12-28 14:53:03 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001708088s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [X...X....................X...........X......................] duration:
6.042330976s trace:
https://jaeger.dev.storj.io/trace/70ba9ccd98ccb34e
FAIL (2025-12-28 14:52:56 UTC)
orbiter warning: context deadline exceeded
Duration: 6.042358358s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [.....XX.XX.X.X..XXXXX....X...X..X..X...X...X.X.........X....] duration:
5.993274232s trace:
https://jaeger.dev.storj.io/trace/550d1e289028a496
FAIL (2025-12-28 14:52:34 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 28 14:49:26.847085842 (551h20m26.718168877s) [upload=483402]: Starting upload
Dec 28 14:49:26.847246297 (551h20m26.718329332s) [upload=483402]: Done starting upload
Dec 28 14:49:26.847480553 (551h20m26.718563588s) [upload=483402]: Uploading segments...
Dec 28 14:49:26.847534546 (551h20m26.718617581s) [upload=483402]: Got next segment. Inline: false
Dec 28 14:49:26.847611382 (551h20m26.718694418s) [upload=483402]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 14:49:26.847625389 (551h20m26.718708424s) [upload=483402]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 14:49:27.094997497 (551h20m26.966080542s) [seg_pos={0 0}, upload=483402]: Begin upload segment...
Dec 28 14:49:27.095052100 (551h20m26.966135145s) [seg_pos={0 0}, upload=483402]: Done begin upload segment.
Dec 28 14:49:27.095090593 (551h20m26.966173638s) [noise=true , address=45.91.116.13:4156 , piece_id=SEVVYACKWMXC6I6QQ54JHPNJBTDE3E2V2J7KFL5WTI7QKHGY4YQQ, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.095108959 (551h20m26.966191994s) [noise=true , address=145.79.227.48:4303 , piece_id=YL7JSA5QHJ3GK4NU344BNUIF5TONSV767UTYEE3YMMF4P7BQ35TA, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.095613940 (551h20m26.966696975s) [noise=true , address=145.79.227.39:4015 , piece_id=TXO5233BTSNH5F4KXE62VOFKV3A5WEFLJBOM72L5ROU6C4M457XQ, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.095909944 (551h20m26.966992979s) [noise=true , address=145.79.227.4:4186 , piece_id=E5RL63ZF2X67454ITCBCP77C3RBY5S5CH4HJQT6EDUAAG4K3MVFQ, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.096198974 (551h20m26.967282009s) [noise=true , address=145.79.227.42:4123 , piece_id=KBOYHLQK2JYAAAD7U3QFQ6HP54KTQPIB4JZF2RV4H7BHLE3WPCZQ, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.096496671 (551h20m26.967579706s) [noise=true , address=45.91.116.6:4976 , piece_id=MJ6RW6NMXBDYGUM2IOBH7HNVFVXSDEJL5UFZKP4FODDP6R6G5OYQ, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.096817342 (551h20m26.967900377s) [noise=true , address=145.79.227.19:4077 , piece_id=VU7DE2EBAGLW7STPZJMH3IPR6FXVOQTXUFWTUH7KSLQIBOVS6CRA, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.097201153 (551h20m26.968284188s) [noise=true , address=145.79.227.6:4259 , piece_id=6OUQWEJO3JCVK4R2CSXHI2USMJMVPWYQSCGYYHUKOA3WOJA2XKCQ, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.097541781 (551h20m26.968624826s) [noise=true , address=145.79.227.37:4566 , piece_id=6RH6NXC5R3U25RAQOHDS42KBJ6QWCS24VRCW6GTKTIIG2FFPHGFQ, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.097932225 (551h20m26.96901526s) [noise=true , address=77.90.31.18:4805 , piece_id=7LO5YQSS2UBZIGEZS7F226PPVMTGLMWKFDDUE2A7WAB3766ILBQA, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.098618392 (551h20m26.969701437s) [noise=true , address=45.91.116.5:4297 , piece_id=75ZWT6KXOSDEE4XG44JHCBU4X5TYYY4FYQ5VHIZSHMGDUBQLRRUQ, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.098971544 (551h20m26.970054579s) [noise=true , address=145.79.227.2:4114 , piece_id=VHSQUG2VEO2ZQZ3DHNLNWYTS4YYHFF35FL7KWWU37GCTAZPLMXUQ, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.099341618 (551h20m26.970424653s) [noise=true , address=45.91.116.8:4048 , piece_id=3PB5EN5FVSYQZ4HUJNXA57A4HRTAQP2GQXECS7MRSFRC7CHXQ7KA, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.099693468 (551h20m26.970776503s) [noise=true , address=145.79.227.25:4619 , piece_id=K56Z6MMBFPBQ5C4ZTZ2JWY6N7FQFEQ6Z7PBDOYL6E5CIFG4MA6LA, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.100159947 (551h20m26.971242992s) [noise=true , address=145.79.227.36:4519 , piece_id=JA3NZNENAPXRYQ65A5RJ4AMOYMIQ75F4KTGH4P6GV3YO4PXXCS4Q, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.100510023 (551h20m26.971593058s) [noise=true , address=145.79.227.50:4373 , piece_id=SGLAGZDBNWO6GTYBAD7257DKCXROXZUDN2RTRFX55AY5UVGTX3RQ, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:28.811420760 (551h20m28.682503805s) [upload=483402]: Next returned nil segment
Dec 28 14:49:28.811513175 (551h20m28.68259621s) [upload=483402]: Waiting for error group managing segments...
Dec 28 14:49:29.101125205 (551h20m28.97220825s) [noise=true , address=145.79.227.25:4619 , piece_id=K56Z6MMBFPBQ5C4ZTZ2JWY6N7FQFEQ6Z7PBDOYL6E5CIFG4MA6LA, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.102240679 (551h20m28.973323724s) [noise=true , address=145.79.227.36:4519 , piece_id=JA3NZNENAPXRYQ65A5RJ4AMOYMIQ75F4KTGH4P6GV3YO4PXXCS4Q, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.103529042 (551h20m28.974612097s) [noise=true , address=145.79.227.19:4077 , piece_id=VU7DE2EBAGLW7STPZJMH3IPR6FXVOQTXUFWTUH7KSLQIBOVS6CRA, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.104590202 (551h20m28.975673247s) [noise=true , address=145.79.227.4:4186 , piece_id=E5RL63ZF2X67454ITCBCP77C3RBY5S5CH4HJQT6EDUAAG4K3MVFQ, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.123887168 (551h20m28.994970203s) [noise=true , address=145.79.227.6:4259 , piece_id=6OUQWEJO3JCVK4R2CSXHI2USMJMVPWYQSCGYYHUKOA3WOJA2XKCQ, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.124475989 (551h20m28.995559054s) [noise=true , address=145.79.227.39:4015 , piece_id=TXO5233BTSNH5F4KXE62VOFKV3A5WEFLJBOM72L5ROU6C4M457XQ, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.124745742 (551h20m28.995828787s) [noise=true , address=145.79.227.37:4566 , piece_id=6RH6NXC5R3U25RAQOHDS42KBJ6QWCS24VRCW6GTKTIIG2FFPHGFQ, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.125303544 (551h20m28.996386579s) [noise=true , address=145.79.227.2:4114 , piece_id=VHSQUG2VEO2ZQZ3DHNLNWYTS4YYHFF35FL7KWWU37GCTAZPLMXUQ, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.128525870 (551h20m28.999608915s) [noise=true , address=145.79.227.50:4373 , piece_id=SGLAGZDBNWO6GTYBAD7257DKCXROXZUDN2RTRFX55AY5UVGTX3RQ, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.130796023 (551h20m29.001879078s) [noise=true , address=145.79.227.42:4123 , piece_id=KBOYHLQK2JYAAAD7U3QFQ6HP54KTQPIB4JZF2RV4H7BHLE3WPCZQ, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.131247102 (551h20m29.002330147s) [noise=true , address=145.79.227.48:4303 , piece_id=YL7JSA5QHJ3GK4NU344BNUIF5TONSV767UTYEE3YMMF4P7BQ35TA, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.313107584 (551h20m29.184190639s) [noise=true , address=45.91.116.6:4976 , piece_id=MJ6RW6NMXBDYGUM2IOBH7HNVFVXSDEJL5UFZKP4FODDP6R6G5OYQ, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.375266024 (551h20m29.246349069s) [noise=true , address=45.91.116.13:4156 , piece_id=SEVVYACKWMXC6I6QQ54JHPNJBTDE3E2V2J7KFL5WTI7QKHGY4YQQ, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.375283747 (551h20m29.246366792s) [seg_pos={0 0}, upload=483402]: Segment reached optimal threshold of 13 pieces.
Dec 28 14:49:29.375478719 (551h20m29.246561754s) [noise=true , address=77.90.31.18:4805 , piece_id=7LO5YQSS2UBZIGEZS7F226PPVMTGLMWKFDDUE2A7WAB3766ILBQA, seg_pos={0 0}, upload=483402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF
Dec 28 14:49:29.375485772 (551h20m29.246568807s) [noise=true , address=77.90.31.18:4805 , piece_id=7LO5YQSS2UBZIGEZS7F226PPVMTGLMWKFDDUE2A7WAB3766ILBQA, seg_pos={0 0}, upload=483402]: Long tail cancellation due to optimal threshold
Dec 28 14:49:29.375489529 (551h20m29.246572564s) [noise=true , address=45.91.116.5:4297 , piece_id=75ZWT6KXOSDEE4XG44JHCBU4X5TYYY4FYQ5VHIZSHMGDUBQLRRUQ, seg_pos={0 0}, upload=483402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF
Dec 28 14:49:29.375498176 (551h20m29.246581231s) [noise=true , address=45.91.116.5:4297 , piece_id=75ZWT6KXOSDEE4XG44JHCBU4X5TYYY4FYQ5VHIZSHMGDUBQLRRUQ, seg_pos={0 0}, upload=483402]: Long tail cancellation due to optimal threshold
Dec 28 14:49:29.375598046 (551h20m29.246681091s) [noise=true , address=45.91.116.8:4048 , piece_id=3PB5EN5FVSYQZ4HUJNXA57A4HRTAQP2GQXECS7MRSFRC7CHXQ7KA, seg_pos={0 0}, upload=483402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF
Dec 28 14:49:29.375603437 (551h20m29.246686472s) [noise=true , address=45.91.116.8:4048 , piece_id=3PB5EN5FVSYQZ4HUJNXA57A4HRTAQP2GQXECS7MRSFRC7CHXQ7KA, seg_pos={0 0}, upload=483402]: Long tail cancellation due to optimal threshold
Dec 28 14:49:29.375617824 (551h20m29.246700859s) [seg_pos={0 0}, upload=483402]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 14:49:29.375687867 (551h20m29.246770912s) [upload=483402]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 14:49:29.375692165 (551h20m29.2467752s) [upload=483402]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 14:49:29.682258328 (551h20m29.553341363s) [upload=483402]: Done uploading segments...
Dec 28 14:49:29.682278987 (551h20m29.553362032s) [upload=483402]: Upload finished. err: <nil>
Duration: 5.99330399s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [.......X...X............X...............X......X....X.......] duration:
47.723601197s trace:
https://jaeger.dev.storj.io/trace/1440a926e26dd799
FAIL (2025-12-28 14:52:33 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
noise=true , address=145.79.227.44:4212 , piece_id=V73IIIU7CCTULBPEXD2YWSFUUGEP2ETVL7LQGIWWEXICZ5SK4BQA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.239195729 (551h16m25.110278774s) [noise=true , address=145.79.227.34:4931 , piece_id=IBNDRK5G5MGWQKRTJ3CS7LI7F3TFUJDHZDG6HJZ44XOQAEWBVEBA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.239549302 (551h16m25.110632347s) [noise=true , address=64.49.13.140:4515 , piece_id=42YNEU3HOVPRPUVOCPKYSQUBTDLBQELQCWNIYJBDXD2RV2NRDJDQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.239966557 (551h16m25.111049602s) [noise=true , address=77.90.31.18:4816 , piece_id=J3DJPEVEESV6EA5KJU6TN7MO4ATQZDBQOSJSH5S5KSZRYGHHZRSQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.240145648 (551h16m25.111228683s) [noise=true , address=77.90.31.14:4688 , piece_id=LNETNIH6PNQ2OR2NQKBWWWOJEFYXIGYHHH34WW7DHTKKYQWJ6DLA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.240933478 (551h16m25.112016523s) [noise=true , address=77.90.31.19:4867 , piece_id=NWQD7PXUGDHUKRF4J3E6EUORCKUM3TDAD2EJH7F3CE55G4EA7SJQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.241389286 (551h16m25.112472331s) [noise=true , address=145.79.227.40:4075 , piece_id=XKJZ2GSNELWKMTMZA5EZ2YTM62XREZQDFI7MDDKAPE3YXRY4KWNA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.241451415 (551h16m25.11253445s) [noise=true , address=77.90.31.21:4889 , piece_id=WSIKD7GCMKHWEFYC7UZ3YWGQHQDIZVW2YJY4BD5ENMCYBJZJMXEQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.241843531 (551h16m25.112926566s) [noise=true , address=145.79.227.46:4608 , piece_id=T5G4T4OU62UPXPIACXJQEZ36O2N63HL4L4HRMW3EN6B4PIE3M7YA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.241911240 (551h16m25.112994285s) [noise=true , address=64.49.13.169:4711 , piece_id=YHY2V4EHCMQEDR6OTZE2POM45Y7FYVNOSEGSUOM5IYWCDWDLRIKQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.242196533 (551h16m25.113279578s) [noise=true , address=193.149.164.143:4697 , piece_id=6SHPKJO6AZLKYPIQWJWNI4YSAMLAVU7BGDGQLQC4I6AGCRWT52DA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.242532343 (551h16m25.113615378s) [noise=true , address=145.79.227.29:4947 , piece_id=FSJZQVSKN76O5RLIVQDPKAN2PDV4VLRCMBRA5EYJGFOETOZ6D7AQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.242582478 (551h16m25.113665613s) [noise=true , address=145.79.227.18:4010 , piece_id=GVY67LYRK3UZRX6OLLWC5JCJEGV3RCIS6DATIDMQCBJSL7DCAE4Q, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.242986888 (551h16m25.114069923s) [noise=true , address=64.49.13.27:4966 , piece_id=N3R7JO5PL47V7ZUXNLXFOHSI2MEJ2SVQNHE6M2DWDYF3D4L34ZCA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.243048475 (551h16m25.11413151s) [noise=true , address=77.90.31.11:4585 , piece_id=5MKOQOD3O7TXKXODJWTVZI5EAKBEO7JE3KKXD63F6SO3EBBSVD6A, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.243336464 (551h16m25.114419499s) [noise=true , address=77.90.31.17:4801 , piece_id=OWQPIGFIL5SFLUI77IYA3LKVUFPSBL6JSJO6XWZLQFV7GXHF3QOA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:26.537548687 (551h16m26.408631722s) [upload=483348]: Next returned nil segment
Dec 28 14:45:26.537559538 (551h16m26.408642573s) [upload=483348]: Waiting for error group managing segments...
Dec 28 14:45:26.899924119 (551h16m26.771007164s) [noise=true , address=64.49.13.140:4515 , piece_id=42YNEU3HOVPRPUVOCPKYSQUBTDLBQELQCWNIYJBDXD2RV2NRDJDQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:26.977957828 (551h16m26.849040873s) [noise=true , address=64.49.13.27:4966 , piece_id=N3R7JO5PL47V7ZUXNLXFOHSI2MEJ2SVQNHE6M2DWDYF3D4L34ZCA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.120995816 (551h16m26.992078862s) [noise=true , address=145.79.227.46:4608 , piece_id=T5G4T4OU62UPXPIACXJQEZ36O2N63HL4L4HRMW3EN6B4PIE3M7YA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.151904071 (551h16m27.022987116s) [noise=true , address=145.79.227.34:4931 , piece_id=IBNDRK5G5MGWQKRTJ3CS7LI7F3TFUJDHZDG6HJZ44XOQAEWBVEBA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.165435146 (551h16m27.036518181s) [noise=true , address=145.79.227.29:4947 , piece_id=FSJZQVSKN76O5RLIVQDPKAN2PDV4VLRCMBRA5EYJGFOETOZ6D7AQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.175130708 (551h16m27.046213754s) [noise=true , address=145.79.227.18:4010 , piece_id=GVY67LYRK3UZRX6OLLWC5JCJEGV3RCIS6DATIDMQCBJSL7DCAE4Q, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.352651587 (551h16m27.223734632s) [noise=true , address=77.90.31.19:4867 , piece_id=NWQD7PXUGDHUKRF4J3E6EUORCKUM3TDAD2EJH7F3CE55G4EA7SJQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.375485928 (551h16m27.246568973s) [noise=true , address=77.90.31.11:4585 , piece_id=5MKOQOD3O7TXKXODJWTVZI5EAKBEO7JE3KKXD63F6SO3EBBSVD6A, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.428947778 (551h16m27.300030823s) [noise=true , address=77.90.31.18:4816 , piece_id=J3DJPEVEESV6EA5KJU6TN7MO4ATQZDBQOSJSH5S5KSZRYGHHZRSQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.567872934 (551h16m27.438955969s) [noise=true , address=77.90.31.17:4801 , piece_id=OWQPIGFIL5SFLUI77IYA3LKVUFPSBL6JSJO6XWZLQFV7GXHF3QOA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.983670843 (551h16m27.854753878s) [noise=true , address=77.90.31.13:4650 , piece_id=NHTWFNKCRHELW7BJ2NR4GGHY4LU743RVLCCR236AO4MQUZD7KMEQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:28.019933813 (551h16m27.891016848s) [noise=true , address=145.79.227.30:4794 , piece_id=OZXLKN4V62NRD3WCMF65C7U7OPM2FPBOIMNNE53LN4MBSDDL65NQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:28.083140990 (551h16m27.954224025s) [noise=true , address=145.79.227.44:4212 , piece_id=V73IIIU7CCTULBPEXD2YWSFUUGEP2ETVL7LQGIWWEXICZ5SK4BQA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:28.129315084 (551h16m28.000398119s) [noise=true , address=145.79.227.40:4075 , piece_id=XKJZ2GSNELWKMTMZA5EZ2YTM62XREZQDFI7MDDKAPE3YXRY4KWNA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:28.129340211 (551h16m28.000423246s) [seg_pos={0 0}, upload=483348]: Segment reached optimal threshold of 14 pieces.
Dec 28 14:45:28.129532938 (551h16m28.000615973s) [noise=true , address=77.90.31.21:4889 , piece_id=WSIKD7GCMKHWEFYC7UZ3YWGQHQDIZVW2YJY4BD5ENMCYBJZJMXEQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wcX8iEhi5LrftcXUQE3Lrm4igNwhDtvT3RNBERTGsKyQ6KLj6): protocol: expected piece hash; context canceled; EOF
Dec 28 14:45:28.129539380 (551h16m28.000622425s) [noise=true , address=77.90.31.21:4889 , piece_id=WSIKD7GCMKHWEFYC7UZ3YWGQHQDIZVW2YJY4BD5ENMCYBJZJMXEQ, seg_pos={0 0}, upload=483348]: Long tail cancellation due to optimal threshold
Dec 28 14:45:28.129666382 (551h16m28.000749418s) [noise=true , address=64.49.13.169:4711 , piece_id=YHY2V4EHCMQEDR6OTZE2POM45Y7FYVNOSEGSUOM5IYWCDWDLRIKQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:47258->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:47258->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47258->64.49.13.169:4711: use of closed network connection
Dec 28 14:45:28.129671853 (551h16m28.000754898s) [noise=true , address=64.49.13.169:4711 , piece_id=YHY2V4EHCMQEDR6OTZE2POM45Y7FYVNOSEGSUOM5IYWCDWDLRIKQ, seg_pos={0 0}, upload=483348]: Long tail cancellation due to optimal threshold
Dec 28 14:45:28.129713071 (551h16m28.000796116s) [noise=true , address=77.90.31.14:4688 , piece_id=LNETNIH6PNQ2OR2NQKBWWWOJEFYXIGYHHH34WW7DHTKKYQWJ6DLA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz): protocol: expected piece hash; context canceled; EOF
Dec 28 14:45:28.129770090 (551h16m28.000853125s) [noise=true , address=77.90.31.14:4688 , piece_id=LNETNIH6PNQ2OR2NQKBWWWOJEFYXIGYHHH34WW7DHTKKYQWJ6DLA, seg_pos={0 0}, upload=483348]: Long tail cancellation due to optimal threshold
Dec 28 14:45:28.129862666 (551h16m28.000945711s) [noise=true , address=193.149.164.143:4697 , piece_id=6SHPKJO6AZLKYPIQWJWNI4YSAMLAVU7BGDGQLQC4I6AGCRWT52DA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ): protocol: expected piece hash; context canceled; EOF
Dec 28 14:45:28.129870651 (551h16m28.000953686s) [noise=true , address=193.149.164.143:4697 , piece_id=6SHPKJO6AZLKYPIQWJWNI4YSAMLAVU7BGDGQLQC4I6AGCRWT52DA, seg_pos={0 0}, upload=483348]: Long tail cancellation due to optimal threshold
Dec 28 14:45:28.129970341 (551h16m28.001053376s) [seg_pos={0 0}, upload=483348]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 14:45:28.130053620 (551h16m28.001136665s) [upload=483348]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 14:45:28.130060553 (551h16m28.001143588s) [upload=483348]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 14:45:28.417819022 (551h16m28.288902057s) [upload=483348]: Done uploading segments...
Dec 28 14:45:28.417844872 (551h16m28.288927927s) [upload=483348]: Upload finished. err: <nil>
Duration: 47.723616345s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
7.447809006s trace:
https://jaeger.dev.storj.io/trace/35a5d4335e2cde85
FAIL (2025-12-28 14:52:33 UTC)
orbiter warning: context deadline exceeded
Duration: 7.447839756s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [.X...X....................X...........X.....................] duration:
12.895583266s trace:
https://jaeger.dev.storj.io/trace/70ba9ccd98ccb34e
FAIL (2025-12-28 14:52:16 UTC)
orbiter warning: context deadline exceeded
Duration: 12.89561649s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................] duration:
16.110153845s trace:
https://jaeger.dev.storj.io/trace/35a5d4335e2cde85
FAIL (2025-12-28 14:52:05 UTC)
orbiter warning: context deadline exceeded
Duration: 16.110178321s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
49 out of 60 is failed (threshold=6) [XXXXX.X.XXXXXXXXXXXXXXXXXXXXXXX.X.XX.XXXXXXXX.X..XXXXX.X.X.X] duration:
5.001596966s trace:
https://jaeger.dev.storj.io/trace/63fea26a1499f6ff
FAIL (2025-12-28 14:52:01 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001626313s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [........X...X............X...............X......X....X......] duration:
58.431037613s trace:
https://jaeger.dev.storj.io/trace/1440a926e26dd799
FAIL (2025-12-28 14:51:45 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
noise=true , address=145.79.227.44:4212 , piece_id=V73IIIU7CCTULBPEXD2YWSFUUGEP2ETVL7LQGIWWEXICZ5SK4BQA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.239195729 (551h16m25.110278774s) [noise=true , address=145.79.227.34:4931 , piece_id=IBNDRK5G5MGWQKRTJ3CS7LI7F3TFUJDHZDG6HJZ44XOQAEWBVEBA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.239549302 (551h16m25.110632347s) [noise=true , address=64.49.13.140:4515 , piece_id=42YNEU3HOVPRPUVOCPKYSQUBTDLBQELQCWNIYJBDXD2RV2NRDJDQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.239966557 (551h16m25.111049602s) [noise=true , address=77.90.31.18:4816 , piece_id=J3DJPEVEESV6EA5KJU6TN7MO4ATQZDBQOSJSH5S5KSZRYGHHZRSQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.240145648 (551h16m25.111228683s) [noise=true , address=77.90.31.14:4688 , piece_id=LNETNIH6PNQ2OR2NQKBWWWOJEFYXIGYHHH34WW7DHTKKYQWJ6DLA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.240933478 (551h16m25.112016523s) [noise=true , address=77.90.31.19:4867 , piece_id=NWQD7PXUGDHUKRF4J3E6EUORCKUM3TDAD2EJH7F3CE55G4EA7SJQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.241389286 (551h16m25.112472331s) [noise=true , address=145.79.227.40:4075 , piece_id=XKJZ2GSNELWKMTMZA5EZ2YTM62XREZQDFI7MDDKAPE3YXRY4KWNA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.241451415 (551h16m25.11253445s) [noise=true , address=77.90.31.21:4889 , piece_id=WSIKD7GCMKHWEFYC7UZ3YWGQHQDIZVW2YJY4BD5ENMCYBJZJMXEQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.241843531 (551h16m25.112926566s) [noise=true , address=145.79.227.46:4608 , piece_id=T5G4T4OU62UPXPIACXJQEZ36O2N63HL4L4HRMW3EN6B4PIE3M7YA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.241911240 (551h16m25.112994285s) [noise=true , address=64.49.13.169:4711 , piece_id=YHY2V4EHCMQEDR6OTZE2POM45Y7FYVNOSEGSUOM5IYWCDWDLRIKQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.242196533 (551h16m25.113279578s) [noise=true , address=193.149.164.143:4697 , piece_id=6SHPKJO6AZLKYPIQWJWNI4YSAMLAVU7BGDGQLQC4I6AGCRWT52DA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.242532343 (551h16m25.113615378s) [noise=true , address=145.79.227.29:4947 , piece_id=FSJZQVSKN76O5RLIVQDPKAN2PDV4VLRCMBRA5EYJGFOETOZ6D7AQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.242582478 (551h16m25.113665613s) [noise=true , address=145.79.227.18:4010 , piece_id=GVY67LYRK3UZRX6OLLWC5JCJEGV3RCIS6DATIDMQCBJSL7DCAE4Q, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.242986888 (551h16m25.114069923s) [noise=true , address=64.49.13.27:4966 , piece_id=N3R7JO5PL47V7ZUXNLXFOHSI2MEJ2SVQNHE6M2DWDYF3D4L34ZCA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.243048475 (551h16m25.11413151s) [noise=true , address=77.90.31.11:4585 , piece_id=5MKOQOD3O7TXKXODJWTVZI5EAKBEO7JE3KKXD63F6SO3EBBSVD6A, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.243336464 (551h16m25.114419499s) [noise=true , address=77.90.31.17:4801 , piece_id=OWQPIGFIL5SFLUI77IYA3LKVUFPSBL6JSJO6XWZLQFV7GXHF3QOA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:26.537548687 (551h16m26.408631722s) [upload=483348]: Next returned nil segment
Dec 28 14:45:26.537559538 (551h16m26.408642573s) [upload=483348]: Waiting for error group managing segments...
Dec 28 14:45:26.899924119 (551h16m26.771007164s) [noise=true , address=64.49.13.140:4515 , piece_id=42YNEU3HOVPRPUVOCPKYSQUBTDLBQELQCWNIYJBDXD2RV2NRDJDQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:26.977957828 (551h16m26.849040873s) [noise=true , address=64.49.13.27:4966 , piece_id=N3R7JO5PL47V7ZUXNLXFOHSI2MEJ2SVQNHE6M2DWDYF3D4L34ZCA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.120995816 (551h16m26.992078862s) [noise=true , address=145.79.227.46:4608 , piece_id=T5G4T4OU62UPXPIACXJQEZ36O2N63HL4L4HRMW3EN6B4PIE3M7YA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.151904071 (551h16m27.022987116s) [noise=true , address=145.79.227.34:4931 , piece_id=IBNDRK5G5MGWQKRTJ3CS7LI7F3TFUJDHZDG6HJZ44XOQAEWBVEBA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.165435146 (551h16m27.036518181s) [noise=true , address=145.79.227.29:4947 , piece_id=FSJZQVSKN76O5RLIVQDPKAN2PDV4VLRCMBRA5EYJGFOETOZ6D7AQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.175130708 (551h16m27.046213754s) [noise=true , address=145.79.227.18:4010 , piece_id=GVY67LYRK3UZRX6OLLWC5JCJEGV3RCIS6DATIDMQCBJSL7DCAE4Q, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.352651587 (551h16m27.223734632s) [noise=true , address=77.90.31.19:4867 , piece_id=NWQD7PXUGDHUKRF4J3E6EUORCKUM3TDAD2EJH7F3CE55G4EA7SJQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.375485928 (551h16m27.246568973s) [noise=true , address=77.90.31.11:4585 , piece_id=5MKOQOD3O7TXKXODJWTVZI5EAKBEO7JE3KKXD63F6SO3EBBSVD6A, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.428947778 (551h16m27.300030823s) [noise=true , address=77.90.31.18:4816 , piece_id=J3DJPEVEESV6EA5KJU6TN7MO4ATQZDBQOSJSH5S5KSZRYGHHZRSQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.567872934 (551h16m27.438955969s) [noise=true , address=77.90.31.17:4801 , piece_id=OWQPIGFIL5SFLUI77IYA3LKVUFPSBL6JSJO6XWZLQFV7GXHF3QOA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.983670843 (551h16m27.854753878s) [noise=true , address=77.90.31.13:4650 , piece_id=NHTWFNKCRHELW7BJ2NR4GGHY4LU743RVLCCR236AO4MQUZD7KMEQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:28.019933813 (551h16m27.891016848s) [noise=true , address=145.79.227.30:4794 , piece_id=OZXLKN4V62NRD3WCMF65C7U7OPM2FPBOIMNNE53LN4MBSDDL65NQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:28.083140990 (551h16m27.954224025s) [noise=true , address=145.79.227.44:4212 , piece_id=V73IIIU7CCTULBPEXD2YWSFUUGEP2ETVL7LQGIWWEXICZ5SK4BQA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:28.129315084 (551h16m28.000398119s) [noise=true , address=145.79.227.40:4075 , piece_id=XKJZ2GSNELWKMTMZA5EZ2YTM62XREZQDFI7MDDKAPE3YXRY4KWNA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:28.129340211 (551h16m28.000423246s) [seg_pos={0 0}, upload=483348]: Segment reached optimal threshold of 14 pieces.
Dec 28 14:45:28.129532938 (551h16m28.000615973s) [noise=true , address=77.90.31.21:4889 , piece_id=WSIKD7GCMKHWEFYC7UZ3YWGQHQDIZVW2YJY4BD5ENMCYBJZJMXEQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wcX8iEhi5LrftcXUQE3Lrm4igNwhDtvT3RNBERTGsKyQ6KLj6): protocol: expected piece hash; context canceled; EOF
Dec 28 14:45:28.129539380 (551h16m28.000622425s) [noise=true , address=77.90.31.21:4889 , piece_id=WSIKD7GCMKHWEFYC7UZ3YWGQHQDIZVW2YJY4BD5ENMCYBJZJMXEQ, seg_pos={0 0}, upload=483348]: Long tail cancellation due to optimal threshold
Dec 28 14:45:28.129666382 (551h16m28.000749418s) [noise=true , address=64.49.13.169:4711 , piece_id=YHY2V4EHCMQEDR6OTZE2POM45Y7FYVNOSEGSUOM5IYWCDWDLRIKQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:47258->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:47258->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47258->64.49.13.169:4711: use of closed network connection
Dec 28 14:45:28.129671853 (551h16m28.000754898s) [noise=true , address=64.49.13.169:4711 , piece_id=YHY2V4EHCMQEDR6OTZE2POM45Y7FYVNOSEGSUOM5IYWCDWDLRIKQ, seg_pos={0 0}, upload=483348]: Long tail cancellation due to optimal threshold
Dec 28 14:45:28.129713071 (551h16m28.000796116s) [noise=true , address=77.90.31.14:4688 , piece_id=LNETNIH6PNQ2OR2NQKBWWWOJEFYXIGYHHH34WW7DHTKKYQWJ6DLA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz): protocol: expected piece hash; context canceled; EOF
Dec 28 14:45:28.129770090 (551h16m28.000853125s) [noise=true , address=77.90.31.14:4688 , piece_id=LNETNIH6PNQ2OR2NQKBWWWOJEFYXIGYHHH34WW7DHTKKYQWJ6DLA, seg_pos={0 0}, upload=483348]: Long tail cancellation due to optimal threshold
Dec 28 14:45:28.129862666 (551h16m28.000945711s) [noise=true , address=193.149.164.143:4697 , piece_id=6SHPKJO6AZLKYPIQWJWNI4YSAMLAVU7BGDGQLQC4I6AGCRWT52DA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ): protocol: expected piece hash; context canceled; EOF
Dec 28 14:45:28.129870651 (551h16m28.000953686s) [noise=true , address=193.149.164.143:4697 , piece_id=6SHPKJO6AZLKYPIQWJWNI4YSAMLAVU7BGDGQLQC4I6AGCRWT52DA, seg_pos={0 0}, upload=483348]: Long tail cancellation due to optimal threshold
Dec 28 14:45:28.129970341 (551h16m28.001053376s) [seg_pos={0 0}, upload=483348]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 14:45:28.130053620 (551h16m28.001136665s) [upload=483348]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 14:45:28.130060553 (551h16m28.001143588s) [upload=483348]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 14:45:28.417819022 (551h16m28.288902057s) [upload=483348]: Done uploading segments...
Dec 28 14:45:28.417844872 (551h16m28.288927927s) [upload=483348]: Upload finished. err: <nil>
Duration: 58.43106753s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [..X...X....................X...........X....................] duration:
9.87359334s trace:
https://jaeger.dev.storj.io/trace/70ba9ccd98ccb34e
FAIL (2025-12-28 14:51:40 UTC)
orbiter warning: context deadline exceeded
Duration: 9.873621143s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [......XX.XX.X.X..XXXXX....X...X..X..X...X...X.X.........X...] duration:
5.712399068s trace:
https://jaeger.dev.storj.io/trace/550d1e289028a496
FAIL (2025-12-28 14:51:38 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 28 14:49:26.847085842 (551h20m26.718168877s) [upload=483402]: Starting upload
Dec 28 14:49:26.847246297 (551h20m26.718329332s) [upload=483402]: Done starting upload
Dec 28 14:49:26.847480553 (551h20m26.718563588s) [upload=483402]: Uploading segments...
Dec 28 14:49:26.847534546 (551h20m26.718617581s) [upload=483402]: Got next segment. Inline: false
Dec 28 14:49:26.847611382 (551h20m26.718694418s) [upload=483402]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 14:49:26.847625389 (551h20m26.718708424s) [upload=483402]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 14:49:27.094997497 (551h20m26.966080542s) [seg_pos={0 0}, upload=483402]: Begin upload segment...
Dec 28 14:49:27.095052100 (551h20m26.966135145s) [seg_pos={0 0}, upload=483402]: Done begin upload segment.
Dec 28 14:49:27.095090593 (551h20m26.966173638s) [noise=true , address=45.91.116.13:4156 , piece_id=SEVVYACKWMXC6I6QQ54JHPNJBTDE3E2V2J7KFL5WTI7QKHGY4YQQ, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.095108959 (551h20m26.966191994s) [noise=true , address=145.79.227.48:4303 , piece_id=YL7JSA5QHJ3GK4NU344BNUIF5TONSV767UTYEE3YMMF4P7BQ35TA, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.095613940 (551h20m26.966696975s) [noise=true , address=145.79.227.39:4015 , piece_id=TXO5233BTSNH5F4KXE62VOFKV3A5WEFLJBOM72L5ROU6C4M457XQ, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.095909944 (551h20m26.966992979s) [noise=true , address=145.79.227.4:4186 , piece_id=E5RL63ZF2X67454ITCBCP77C3RBY5S5CH4HJQT6EDUAAG4K3MVFQ, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.096198974 (551h20m26.967282009s) [noise=true , address=145.79.227.42:4123 , piece_id=KBOYHLQK2JYAAAD7U3QFQ6HP54KTQPIB4JZF2RV4H7BHLE3WPCZQ, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.096496671 (551h20m26.967579706s) [noise=true , address=45.91.116.6:4976 , piece_id=MJ6RW6NMXBDYGUM2IOBH7HNVFVXSDEJL5UFZKP4FODDP6R6G5OYQ, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.096817342 (551h20m26.967900377s) [noise=true , address=145.79.227.19:4077 , piece_id=VU7DE2EBAGLW7STPZJMH3IPR6FXVOQTXUFWTUH7KSLQIBOVS6CRA, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.097201153 (551h20m26.968284188s) [noise=true , address=145.79.227.6:4259 , piece_id=6OUQWEJO3JCVK4R2CSXHI2USMJMVPWYQSCGYYHUKOA3WOJA2XKCQ, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.097541781 (551h20m26.968624826s) [noise=true , address=145.79.227.37:4566 , piece_id=6RH6NXC5R3U25RAQOHDS42KBJ6QWCS24VRCW6GTKTIIG2FFPHGFQ, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.097932225 (551h20m26.96901526s) [noise=true , address=77.90.31.18:4805 , piece_id=7LO5YQSS2UBZIGEZS7F226PPVMTGLMWKFDDUE2A7WAB3766ILBQA, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.098618392 (551h20m26.969701437s) [noise=true , address=45.91.116.5:4297 , piece_id=75ZWT6KXOSDEE4XG44JHCBU4X5TYYY4FYQ5VHIZSHMGDUBQLRRUQ, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.098971544 (551h20m26.970054579s) [noise=true , address=145.79.227.2:4114 , piece_id=VHSQUG2VEO2ZQZ3DHNLNWYTS4YYHFF35FL7KWWU37GCTAZPLMXUQ, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.099341618 (551h20m26.970424653s) [noise=true , address=45.91.116.8:4048 , piece_id=3PB5EN5FVSYQZ4HUJNXA57A4HRTAQP2GQXECS7MRSFRC7CHXQ7KA, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.099693468 (551h20m26.970776503s) [noise=true , address=145.79.227.25:4619 , piece_id=K56Z6MMBFPBQ5C4ZTZ2JWY6N7FQFEQ6Z7PBDOYL6E5CIFG4MA6LA, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.100159947 (551h20m26.971242992s) [noise=true , address=145.79.227.36:4519 , piece_id=JA3NZNENAPXRYQ65A5RJ4AMOYMIQ75F4KTGH4P6GV3YO4PXXCS4Q, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.100510023 (551h20m26.971593058s) [noise=true , address=145.79.227.50:4373 , piece_id=SGLAGZDBNWO6GTYBAD7257DKCXROXZUDN2RTRFX55AY5UVGTX3RQ, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:28.811420760 (551h20m28.682503805s) [upload=483402]: Next returned nil segment
Dec 28 14:49:28.811513175 (551h20m28.68259621s) [upload=483402]: Waiting for error group managing segments...
Dec 28 14:49:29.101125205 (551h20m28.97220825s) [noise=true , address=145.79.227.25:4619 , piece_id=K56Z6MMBFPBQ5C4ZTZ2JWY6N7FQFEQ6Z7PBDOYL6E5CIFG4MA6LA, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.102240679 (551h20m28.973323724s) [noise=true , address=145.79.227.36:4519 , piece_id=JA3NZNENAPXRYQ65A5RJ4AMOYMIQ75F4KTGH4P6GV3YO4PXXCS4Q, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.103529042 (551h20m28.974612097s) [noise=true , address=145.79.227.19:4077 , piece_id=VU7DE2EBAGLW7STPZJMH3IPR6FXVOQTXUFWTUH7KSLQIBOVS6CRA, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.104590202 (551h20m28.975673247s) [noise=true , address=145.79.227.4:4186 , piece_id=E5RL63ZF2X67454ITCBCP77C3RBY5S5CH4HJQT6EDUAAG4K3MVFQ, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.123887168 (551h20m28.994970203s) [noise=true , address=145.79.227.6:4259 , piece_id=6OUQWEJO3JCVK4R2CSXHI2USMJMVPWYQSCGYYHUKOA3WOJA2XKCQ, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.124475989 (551h20m28.995559054s) [noise=true , address=145.79.227.39:4015 , piece_id=TXO5233BTSNH5F4KXE62VOFKV3A5WEFLJBOM72L5ROU6C4M457XQ, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.124745742 (551h20m28.995828787s) [noise=true , address=145.79.227.37:4566 , piece_id=6RH6NXC5R3U25RAQOHDS42KBJ6QWCS24VRCW6GTKTIIG2FFPHGFQ, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.125303544 (551h20m28.996386579s) [noise=true , address=145.79.227.2:4114 , piece_id=VHSQUG2VEO2ZQZ3DHNLNWYTS4YYHFF35FL7KWWU37GCTAZPLMXUQ, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.128525870 (551h20m28.999608915s) [noise=true , address=145.79.227.50:4373 , piece_id=SGLAGZDBNWO6GTYBAD7257DKCXROXZUDN2RTRFX55AY5UVGTX3RQ, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.130796023 (551h20m29.001879078s) [noise=true , address=145.79.227.42:4123 , piece_id=KBOYHLQK2JYAAAD7U3QFQ6HP54KTQPIB4JZF2RV4H7BHLE3WPCZQ, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.131247102 (551h20m29.002330147s) [noise=true , address=145.79.227.48:4303 , piece_id=YL7JSA5QHJ3GK4NU344BNUIF5TONSV767UTYEE3YMMF4P7BQ35TA, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.313107584 (551h20m29.184190639s) [noise=true , address=45.91.116.6:4976 , piece_id=MJ6RW6NMXBDYGUM2IOBH7HNVFVXSDEJL5UFZKP4FODDP6R6G5OYQ, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.375266024 (551h20m29.246349069s) [noise=true , address=45.91.116.13:4156 , piece_id=SEVVYACKWMXC6I6QQ54JHPNJBTDE3E2V2J7KFL5WTI7QKHGY4YQQ, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.375283747 (551h20m29.246366792s) [seg_pos={0 0}, upload=483402]: Segment reached optimal threshold of 13 pieces.
Dec 28 14:49:29.375478719 (551h20m29.246561754s) [noise=true , address=77.90.31.18:4805 , piece_id=7LO5YQSS2UBZIGEZS7F226PPVMTGLMWKFDDUE2A7WAB3766ILBQA, seg_pos={0 0}, upload=483402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF
Dec 28 14:49:29.375485772 (551h20m29.246568807s) [noise=true , address=77.90.31.18:4805 , piece_id=7LO5YQSS2UBZIGEZS7F226PPVMTGLMWKFDDUE2A7WAB3766ILBQA, seg_pos={0 0}, upload=483402]: Long tail cancellation due to optimal threshold
Dec 28 14:49:29.375489529 (551h20m29.246572564s) [noise=true , address=45.91.116.5:4297 , piece_id=75ZWT6KXOSDEE4XG44JHCBU4X5TYYY4FYQ5VHIZSHMGDUBQLRRUQ, seg_pos={0 0}, upload=483402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF
Dec 28 14:49:29.375498176 (551h20m29.246581231s) [noise=true , address=45.91.116.5:4297 , piece_id=75ZWT6KXOSDEE4XG44JHCBU4X5TYYY4FYQ5VHIZSHMGDUBQLRRUQ, seg_pos={0 0}, upload=483402]: Long tail cancellation due to optimal threshold
Dec 28 14:49:29.375598046 (551h20m29.246681091s) [noise=true , address=45.91.116.8:4048 , piece_id=3PB5EN5FVSYQZ4HUJNXA57A4HRTAQP2GQXECS7MRSFRC7CHXQ7KA, seg_pos={0 0}, upload=483402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF
Dec 28 14:49:29.375603437 (551h20m29.246686472s) [noise=true , address=45.91.116.8:4048 , piece_id=3PB5EN5FVSYQZ4HUJNXA57A4HRTAQP2GQXECS7MRSFRC7CHXQ7KA, seg_pos={0 0}, upload=483402]: Long tail cancellation due to optimal threshold
Dec 28 14:49:29.375617824 (551h20m29.246700859s) [seg_pos={0 0}, upload=483402]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 14:49:29.375687867 (551h20m29.246770912s) [upload=483402]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 14:49:29.375692165 (551h20m29.2467752s) [upload=483402]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 14:49:29.682258328 (551h20m29.553341363s) [upload=483402]: Done uploading segments...
Dec 28 14:49:29.682278987 (551h20m29.553362032s) [upload=483402]: Upload finished. err: <nil>
Duration: 5.712429417s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
49 out of 60 is failed (threshold=6) [XXXXXX.X.XXXXXXXXXXXXXXXXXXXXXXX.X.XX.XXXXXXXX.X..XXXXX.X.X.] duration:
4.384898647s trace:
https://jaeger.dev.storj.io/trace/1ebed011162f15fd
FAIL (2025-12-28 14:51:17 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.384927161s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............X...............................................] duration:
15.407439694s trace:
https://jaeger.dev.storj.io/trace/35a5d4335e2cde85
FAIL (2025-12-28 14:51:09 UTC)
orbiter warning: context deadline exceeded
Duration: 15.407468329s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [...X...X....................X...........X...................] duration:
14.987024646s trace:
https://jaeger.dev.storj.io/trace/70ba9ccd98ccb34e
FAIL (2025-12-28 14:50:39 UTC)
orbiter warning: context deadline exceeded
Duration: 14.987053933s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
20 out of 60 is failed (threshold=3) [X......XX.XX.X.X..XXXXX....X...X..X..X...X...X.X.........X..] duration:
5.655477632s trace:
https://jaeger.dev.storj.io/trace/550d1e289028a496
FAIL (2025-12-28 14:50:38 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 28 14:49:26.847085842 (551h20m26.718168877s) [upload=483402]: Starting upload
Dec 28 14:49:26.847246297 (551h20m26.718329332s) [upload=483402]: Done starting upload
Dec 28 14:49:26.847480553 (551h20m26.718563588s) [upload=483402]: Uploading segments...
Dec 28 14:49:26.847534546 (551h20m26.718617581s) [upload=483402]: Got next segment. Inline: false
Dec 28 14:49:26.847611382 (551h20m26.718694418s) [upload=483402]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 14:49:26.847625389 (551h20m26.718708424s) [upload=483402]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 14:49:27.094997497 (551h20m26.966080542s) [seg_pos={0 0}, upload=483402]: Begin upload segment...
Dec 28 14:49:27.095052100 (551h20m26.966135145s) [seg_pos={0 0}, upload=483402]: Done begin upload segment.
Dec 28 14:49:27.095090593 (551h20m26.966173638s) [noise=true , address=45.91.116.13:4156 , piece_id=SEVVYACKWMXC6I6QQ54JHPNJBTDE3E2V2J7KFL5WTI7QKHGY4YQQ, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.095108959 (551h20m26.966191994s) [noise=true , address=145.79.227.48:4303 , piece_id=YL7JSA5QHJ3GK4NU344BNUIF5TONSV767UTYEE3YMMF4P7BQ35TA, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.095613940 (551h20m26.966696975s) [noise=true , address=145.79.227.39:4015 , piece_id=TXO5233BTSNH5F4KXE62VOFKV3A5WEFLJBOM72L5ROU6C4M457XQ, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.095909944 (551h20m26.966992979s) [noise=true , address=145.79.227.4:4186 , piece_id=E5RL63ZF2X67454ITCBCP77C3RBY5S5CH4HJQT6EDUAAG4K3MVFQ, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.096198974 (551h20m26.967282009s) [noise=true , address=145.79.227.42:4123 , piece_id=KBOYHLQK2JYAAAD7U3QFQ6HP54KTQPIB4JZF2RV4H7BHLE3WPCZQ, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.096496671 (551h20m26.967579706s) [noise=true , address=45.91.116.6:4976 , piece_id=MJ6RW6NMXBDYGUM2IOBH7HNVFVXSDEJL5UFZKP4FODDP6R6G5OYQ, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.096817342 (551h20m26.967900377s) [noise=true , address=145.79.227.19:4077 , piece_id=VU7DE2EBAGLW7STPZJMH3IPR6FXVOQTXUFWTUH7KSLQIBOVS6CRA, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.097201153 (551h20m26.968284188s) [noise=true , address=145.79.227.6:4259 , piece_id=6OUQWEJO3JCVK4R2CSXHI2USMJMVPWYQSCGYYHUKOA3WOJA2XKCQ, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.097541781 (551h20m26.968624826s) [noise=true , address=145.79.227.37:4566 , piece_id=6RH6NXC5R3U25RAQOHDS42KBJ6QWCS24VRCW6GTKTIIG2FFPHGFQ, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.097932225 (551h20m26.96901526s) [noise=true , address=77.90.31.18:4805 , piece_id=7LO5YQSS2UBZIGEZS7F226PPVMTGLMWKFDDUE2A7WAB3766ILBQA, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.098618392 (551h20m26.969701437s) [noise=true , address=45.91.116.5:4297 , piece_id=75ZWT6KXOSDEE4XG44JHCBU4X5TYYY4FYQ5VHIZSHMGDUBQLRRUQ, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.098971544 (551h20m26.970054579s) [noise=true , address=145.79.227.2:4114 , piece_id=VHSQUG2VEO2ZQZ3DHNLNWYTS4YYHFF35FL7KWWU37GCTAZPLMXUQ, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.099341618 (551h20m26.970424653s) [noise=true , address=45.91.116.8:4048 , piece_id=3PB5EN5FVSYQZ4HUJNXA57A4HRTAQP2GQXECS7MRSFRC7CHXQ7KA, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.099693468 (551h20m26.970776503s) [noise=true , address=145.79.227.25:4619 , piece_id=K56Z6MMBFPBQ5C4ZTZ2JWY6N7FQFEQ6Z7PBDOYL6E5CIFG4MA6LA, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.100159947 (551h20m26.971242992s) [noise=true , address=145.79.227.36:4519 , piece_id=JA3NZNENAPXRYQ65A5RJ4AMOYMIQ75F4KTGH4P6GV3YO4PXXCS4Q, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.100510023 (551h20m26.971593058s) [noise=true , address=145.79.227.50:4373 , piece_id=SGLAGZDBNWO6GTYBAD7257DKCXROXZUDN2RTRFX55AY5UVGTX3RQ, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:28.811420760 (551h20m28.682503805s) [upload=483402]: Next returned nil segment
Dec 28 14:49:28.811513175 (551h20m28.68259621s) [upload=483402]: Waiting for error group managing segments...
Dec 28 14:49:29.101125205 (551h20m28.97220825s) [noise=true , address=145.79.227.25:4619 , piece_id=K56Z6MMBFPBQ5C4ZTZ2JWY6N7FQFEQ6Z7PBDOYL6E5CIFG4MA6LA, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.102240679 (551h20m28.973323724s) [noise=true , address=145.79.227.36:4519 , piece_id=JA3NZNENAPXRYQ65A5RJ4AMOYMIQ75F4KTGH4P6GV3YO4PXXCS4Q, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.103529042 (551h20m28.974612097s) [noise=true , address=145.79.227.19:4077 , piece_id=VU7DE2EBAGLW7STPZJMH3IPR6FXVOQTXUFWTUH7KSLQIBOVS6CRA, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.104590202 (551h20m28.975673247s) [noise=true , address=145.79.227.4:4186 , piece_id=E5RL63ZF2X67454ITCBCP77C3RBY5S5CH4HJQT6EDUAAG4K3MVFQ, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.123887168 (551h20m28.994970203s) [noise=true , address=145.79.227.6:4259 , piece_id=6OUQWEJO3JCVK4R2CSXHI2USMJMVPWYQSCGYYHUKOA3WOJA2XKCQ, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.124475989 (551h20m28.995559054s) [noise=true , address=145.79.227.39:4015 , piece_id=TXO5233BTSNH5F4KXE62VOFKV3A5WEFLJBOM72L5ROU6C4M457XQ, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.124745742 (551h20m28.995828787s) [noise=true , address=145.79.227.37:4566 , piece_id=6RH6NXC5R3U25RAQOHDS42KBJ6QWCS24VRCW6GTKTIIG2FFPHGFQ, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.125303544 (551h20m28.996386579s) [noise=true , address=145.79.227.2:4114 , piece_id=VHSQUG2VEO2ZQZ3DHNLNWYTS4YYHFF35FL7KWWU37GCTAZPLMXUQ, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.128525870 (551h20m28.999608915s) [noise=true , address=145.79.227.50:4373 , piece_id=SGLAGZDBNWO6GTYBAD7257DKCXROXZUDN2RTRFX55AY5UVGTX3RQ, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.130796023 (551h20m29.001879078s) [noise=true , address=145.79.227.42:4123 , piece_id=KBOYHLQK2JYAAAD7U3QFQ6HP54KTQPIB4JZF2RV4H7BHLE3WPCZQ, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.131247102 (551h20m29.002330147s) [noise=true , address=145.79.227.48:4303 , piece_id=YL7JSA5QHJ3GK4NU344BNUIF5TONSV767UTYEE3YMMF4P7BQ35TA, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.313107584 (551h20m29.184190639s) [noise=true , address=45.91.116.6:4976 , piece_id=MJ6RW6NMXBDYGUM2IOBH7HNVFVXSDEJL5UFZKP4FODDP6R6G5OYQ, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.375266024 (551h20m29.246349069s) [noise=true , address=45.91.116.13:4156 , piece_id=SEVVYACKWMXC6I6QQ54JHPNJBTDE3E2V2J7KFL5WTI7QKHGY4YQQ, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.375283747 (551h20m29.246366792s) [seg_pos={0 0}, upload=483402]: Segment reached optimal threshold of 13 pieces.
Dec 28 14:49:29.375478719 (551h20m29.246561754s) [noise=true , address=77.90.31.18:4805 , piece_id=7LO5YQSS2UBZIGEZS7F226PPVMTGLMWKFDDUE2A7WAB3766ILBQA, seg_pos={0 0}, upload=483402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF
Dec 28 14:49:29.375485772 (551h20m29.246568807s) [noise=true , address=77.90.31.18:4805 , piece_id=7LO5YQSS2UBZIGEZS7F226PPVMTGLMWKFDDUE2A7WAB3766ILBQA, seg_pos={0 0}, upload=483402]: Long tail cancellation due to optimal threshold
Dec 28 14:49:29.375489529 (551h20m29.246572564s) [noise=true , address=45.91.116.5:4297 , piece_id=75ZWT6KXOSDEE4XG44JHCBU4X5TYYY4FYQ5VHIZSHMGDUBQLRRUQ, seg_pos={0 0}, upload=483402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF
Dec 28 14:49:29.375498176 (551h20m29.246581231s) [noise=true , address=45.91.116.5:4297 , piece_id=75ZWT6KXOSDEE4XG44JHCBU4X5TYYY4FYQ5VHIZSHMGDUBQLRRUQ, seg_pos={0 0}, upload=483402]: Long tail cancellation due to optimal threshold
Dec 28 14:49:29.375598046 (551h20m29.246681091s) [noise=true , address=45.91.116.8:4048 , piece_id=3PB5EN5FVSYQZ4HUJNXA57A4HRTAQP2GQXECS7MRSFRC7CHXQ7KA, seg_pos={0 0}, upload=483402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF
Dec 28 14:49:29.375603437 (551h20m29.246686472s) [noise=true , address=45.91.116.8:4048 , piece_id=3PB5EN5FVSYQZ4HUJNXA57A4HRTAQP2GQXECS7MRSFRC7CHXQ7KA, seg_pos={0 0}, upload=483402]: Long tail cancellation due to optimal threshold
Dec 28 14:49:29.375617824 (551h20m29.246700859s) [seg_pos={0 0}, upload=483402]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 14:49:29.375687867 (551h20m29.246770912s) [upload=483402]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 14:49:29.375692165 (551h20m29.2467752s) [upload=483402]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 14:49:29.682258328 (551h20m29.553341363s) [upload=483402]: Done uploading segments...
Dec 28 14:49:29.682278987 (551h20m29.553362032s) [upload=483402]: Upload finished. err: <nil>
Duration: 5.655501868s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
7 out of 60 is failed (threshold=3) [X........X...X............X...............X......X....X.....] duration:
41.412802858s trace:
https://jaeger.dev.storj.io/trace/1440a926e26dd799
FAIL (2025-12-28 14:50:38 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
noise=true , address=145.79.227.44:4212 , piece_id=V73IIIU7CCTULBPEXD2YWSFUUGEP2ETVL7LQGIWWEXICZ5SK4BQA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.239195729 (551h16m25.110278774s) [noise=true , address=145.79.227.34:4931 , piece_id=IBNDRK5G5MGWQKRTJ3CS7LI7F3TFUJDHZDG6HJZ44XOQAEWBVEBA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.239549302 (551h16m25.110632347s) [noise=true , address=64.49.13.140:4515 , piece_id=42YNEU3HOVPRPUVOCPKYSQUBTDLBQELQCWNIYJBDXD2RV2NRDJDQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.239966557 (551h16m25.111049602s) [noise=true , address=77.90.31.18:4816 , piece_id=J3DJPEVEESV6EA5KJU6TN7MO4ATQZDBQOSJSH5S5KSZRYGHHZRSQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.240145648 (551h16m25.111228683s) [noise=true , address=77.90.31.14:4688 , piece_id=LNETNIH6PNQ2OR2NQKBWWWOJEFYXIGYHHH34WW7DHTKKYQWJ6DLA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.240933478 (551h16m25.112016523s) [noise=true , address=77.90.31.19:4867 , piece_id=NWQD7PXUGDHUKRF4J3E6EUORCKUM3TDAD2EJH7F3CE55G4EA7SJQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.241389286 (551h16m25.112472331s) [noise=true , address=145.79.227.40:4075 , piece_id=XKJZ2GSNELWKMTMZA5EZ2YTM62XREZQDFI7MDDKAPE3YXRY4KWNA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.241451415 (551h16m25.11253445s) [noise=true , address=77.90.31.21:4889 , piece_id=WSIKD7GCMKHWEFYC7UZ3YWGQHQDIZVW2YJY4BD5ENMCYBJZJMXEQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.241843531 (551h16m25.112926566s) [noise=true , address=145.79.227.46:4608 , piece_id=T5G4T4OU62UPXPIACXJQEZ36O2N63HL4L4HRMW3EN6B4PIE3M7YA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.241911240 (551h16m25.112994285s) [noise=true , address=64.49.13.169:4711 , piece_id=YHY2V4EHCMQEDR6OTZE2POM45Y7FYVNOSEGSUOM5IYWCDWDLRIKQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.242196533 (551h16m25.113279578s) [noise=true , address=193.149.164.143:4697 , piece_id=6SHPKJO6AZLKYPIQWJWNI4YSAMLAVU7BGDGQLQC4I6AGCRWT52DA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.242532343 (551h16m25.113615378s) [noise=true , address=145.79.227.29:4947 , piece_id=FSJZQVSKN76O5RLIVQDPKAN2PDV4VLRCMBRA5EYJGFOETOZ6D7AQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.242582478 (551h16m25.113665613s) [noise=true , address=145.79.227.18:4010 , piece_id=GVY67LYRK3UZRX6OLLWC5JCJEGV3RCIS6DATIDMQCBJSL7DCAE4Q, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.242986888 (551h16m25.114069923s) [noise=true , address=64.49.13.27:4966 , piece_id=N3R7JO5PL47V7ZUXNLXFOHSI2MEJ2SVQNHE6M2DWDYF3D4L34ZCA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.243048475 (551h16m25.11413151s) [noise=true , address=77.90.31.11:4585 , piece_id=5MKOQOD3O7TXKXODJWTVZI5EAKBEO7JE3KKXD63F6SO3EBBSVD6A, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.243336464 (551h16m25.114419499s) [noise=true , address=77.90.31.17:4801 , piece_id=OWQPIGFIL5SFLUI77IYA3LKVUFPSBL6JSJO6XWZLQFV7GXHF3QOA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:26.537548687 (551h16m26.408631722s) [upload=483348]: Next returned nil segment
Dec 28 14:45:26.537559538 (551h16m26.408642573s) [upload=483348]: Waiting for error group managing segments...
Dec 28 14:45:26.899924119 (551h16m26.771007164s) [noise=true , address=64.49.13.140:4515 , piece_id=42YNEU3HOVPRPUVOCPKYSQUBTDLBQELQCWNIYJBDXD2RV2NRDJDQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:26.977957828 (551h16m26.849040873s) [noise=true , address=64.49.13.27:4966 , piece_id=N3R7JO5PL47V7ZUXNLXFOHSI2MEJ2SVQNHE6M2DWDYF3D4L34ZCA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.120995816 (551h16m26.992078862s) [noise=true , address=145.79.227.46:4608 , piece_id=T5G4T4OU62UPXPIACXJQEZ36O2N63HL4L4HRMW3EN6B4PIE3M7YA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.151904071 (551h16m27.022987116s) [noise=true , address=145.79.227.34:4931 , piece_id=IBNDRK5G5MGWQKRTJ3CS7LI7F3TFUJDHZDG6HJZ44XOQAEWBVEBA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.165435146 (551h16m27.036518181s) [noise=true , address=145.79.227.29:4947 , piece_id=FSJZQVSKN76O5RLIVQDPKAN2PDV4VLRCMBRA5EYJGFOETOZ6D7AQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.175130708 (551h16m27.046213754s) [noise=true , address=145.79.227.18:4010 , piece_id=GVY67LYRK3UZRX6OLLWC5JCJEGV3RCIS6DATIDMQCBJSL7DCAE4Q, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.352651587 (551h16m27.223734632s) [noise=true , address=77.90.31.19:4867 , piece_id=NWQD7PXUGDHUKRF4J3E6EUORCKUM3TDAD2EJH7F3CE55G4EA7SJQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.375485928 (551h16m27.246568973s) [noise=true , address=77.90.31.11:4585 , piece_id=5MKOQOD3O7TXKXODJWTVZI5EAKBEO7JE3KKXD63F6SO3EBBSVD6A, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.428947778 (551h16m27.300030823s) [noise=true , address=77.90.31.18:4816 , piece_id=J3DJPEVEESV6EA5KJU6TN7MO4ATQZDBQOSJSH5S5KSZRYGHHZRSQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.567872934 (551h16m27.438955969s) [noise=true , address=77.90.31.17:4801 , piece_id=OWQPIGFIL5SFLUI77IYA3LKVUFPSBL6JSJO6XWZLQFV7GXHF3QOA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.983670843 (551h16m27.854753878s) [noise=true , address=77.90.31.13:4650 , piece_id=NHTWFNKCRHELW7BJ2NR4GGHY4LU743RVLCCR236AO4MQUZD7KMEQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:28.019933813 (551h16m27.891016848s) [noise=true , address=145.79.227.30:4794 , piece_id=OZXLKN4V62NRD3WCMF65C7U7OPM2FPBOIMNNE53LN4MBSDDL65NQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:28.083140990 (551h16m27.954224025s) [noise=true , address=145.79.227.44:4212 , piece_id=V73IIIU7CCTULBPEXD2YWSFUUGEP2ETVL7LQGIWWEXICZ5SK4BQA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:28.129315084 (551h16m28.000398119s) [noise=true , address=145.79.227.40:4075 , piece_id=XKJZ2GSNELWKMTMZA5EZ2YTM62XREZQDFI7MDDKAPE3YXRY4KWNA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:28.129340211 (551h16m28.000423246s) [seg_pos={0 0}, upload=483348]: Segment reached optimal threshold of 14 pieces.
Dec 28 14:45:28.129532938 (551h16m28.000615973s) [noise=true , address=77.90.31.21:4889 , piece_id=WSIKD7GCMKHWEFYC7UZ3YWGQHQDIZVW2YJY4BD5ENMCYBJZJMXEQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wcX8iEhi5LrftcXUQE3Lrm4igNwhDtvT3RNBERTGsKyQ6KLj6): protocol: expected piece hash; context canceled; EOF
Dec 28 14:45:28.129539380 (551h16m28.000622425s) [noise=true , address=77.90.31.21:4889 , piece_id=WSIKD7GCMKHWEFYC7UZ3YWGQHQDIZVW2YJY4BD5ENMCYBJZJMXEQ, seg_pos={0 0}, upload=483348]: Long tail cancellation due to optimal threshold
Dec 28 14:45:28.129666382 (551h16m28.000749418s) [noise=true , address=64.49.13.169:4711 , piece_id=YHY2V4EHCMQEDR6OTZE2POM45Y7FYVNOSEGSUOM5IYWCDWDLRIKQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:47258->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:47258->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47258->64.49.13.169:4711: use of closed network connection
Dec 28 14:45:28.129671853 (551h16m28.000754898s) [noise=true , address=64.49.13.169:4711 , piece_id=YHY2V4EHCMQEDR6OTZE2POM45Y7FYVNOSEGSUOM5IYWCDWDLRIKQ, seg_pos={0 0}, upload=483348]: Long tail cancellation due to optimal threshold
Dec 28 14:45:28.129713071 (551h16m28.000796116s) [noise=true , address=77.90.31.14:4688 , piece_id=LNETNIH6PNQ2OR2NQKBWWWOJEFYXIGYHHH34WW7DHTKKYQWJ6DLA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz): protocol: expected piece hash; context canceled; EOF
Dec 28 14:45:28.129770090 (551h16m28.000853125s) [noise=true , address=77.90.31.14:4688 , piece_id=LNETNIH6PNQ2OR2NQKBWWWOJEFYXIGYHHH34WW7DHTKKYQWJ6DLA, seg_pos={0 0}, upload=483348]: Long tail cancellation due to optimal threshold
Dec 28 14:45:28.129862666 (551h16m28.000945711s) [noise=true , address=193.149.164.143:4697 , piece_id=6SHPKJO6AZLKYPIQWJWNI4YSAMLAVU7BGDGQLQC4I6AGCRWT52DA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ): protocol: expected piece hash; context canceled; EOF
Dec 28 14:45:28.129870651 (551h16m28.000953686s) [noise=true , address=193.149.164.143:4697 , piece_id=6SHPKJO6AZLKYPIQWJWNI4YSAMLAVU7BGDGQLQC4I6AGCRWT52DA, seg_pos={0 0}, upload=483348]: Long tail cancellation due to optimal threshold
Dec 28 14:45:28.129970341 (551h16m28.001053376s) [seg_pos={0 0}, upload=483348]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 14:45:28.130053620 (551h16m28.001136665s) [upload=483348]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 14:45:28.130060553 (551h16m28.001143588s) [upload=483348]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 14:45:28.417819022 (551h16m28.288902057s) [upload=483348]: Done uploading segments...
Dec 28 14:45:28.417844872 (551h16m28.288927927s) [upload=483348]: Upload finished. err: <nil>
Duration: 41.412817426s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
50 out of 60 is failed (threshold=6) [XXXXXXX.X.XXXXXXXXXXXXXXXXXXXXXXX.X.XX.XXXXXXXX.X..XXXXX.X.X] duration:
5.001686355s trace:
https://jaeger.dev.storj.io/trace/1ebed011162f15fd
FAIL (2025-12-28 14:50:07 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001711672s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
21 out of 60 is failed (threshold=3) [XX......XX.XX.X.X..XXXXX....X...X..X..X...X...X.X.........X.] duration:
6.004828495s trace:
https://jaeger.dev.storj.io/trace/550d1e289028a496
FAIL (2025-12-28 14:50:06 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 28 14:49:26.847085842 (551h20m26.718168877s) [upload=483402]: Starting upload
Dec 28 14:49:26.847246297 (551h20m26.718329332s) [upload=483402]: Done starting upload
Dec 28 14:49:26.847480553 (551h20m26.718563588s) [upload=483402]: Uploading segments...
Dec 28 14:49:26.847534546 (551h20m26.718617581s) [upload=483402]: Got next segment. Inline: false
Dec 28 14:49:26.847611382 (551h20m26.718694418s) [upload=483402]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 14:49:26.847625389 (551h20m26.718708424s) [upload=483402]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 14:49:27.094997497 (551h20m26.966080542s) [seg_pos={0 0}, upload=483402]: Begin upload segment...
Dec 28 14:49:27.095052100 (551h20m26.966135145s) [seg_pos={0 0}, upload=483402]: Done begin upload segment.
Dec 28 14:49:27.095090593 (551h20m26.966173638s) [noise=true , address=45.91.116.13:4156 , piece_id=SEVVYACKWMXC6I6QQ54JHPNJBTDE3E2V2J7KFL5WTI7QKHGY4YQQ, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.095108959 (551h20m26.966191994s) [noise=true , address=145.79.227.48:4303 , piece_id=YL7JSA5QHJ3GK4NU344BNUIF5TONSV767UTYEE3YMMF4P7BQ35TA, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.095613940 (551h20m26.966696975s) [noise=true , address=145.79.227.39:4015 , piece_id=TXO5233BTSNH5F4KXE62VOFKV3A5WEFLJBOM72L5ROU6C4M457XQ, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.095909944 (551h20m26.966992979s) [noise=true , address=145.79.227.4:4186 , piece_id=E5RL63ZF2X67454ITCBCP77C3RBY5S5CH4HJQT6EDUAAG4K3MVFQ, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.096198974 (551h20m26.967282009s) [noise=true , address=145.79.227.42:4123 , piece_id=KBOYHLQK2JYAAAD7U3QFQ6HP54KTQPIB4JZF2RV4H7BHLE3WPCZQ, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.096496671 (551h20m26.967579706s) [noise=true , address=45.91.116.6:4976 , piece_id=MJ6RW6NMXBDYGUM2IOBH7HNVFVXSDEJL5UFZKP4FODDP6R6G5OYQ, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.096817342 (551h20m26.967900377s) [noise=true , address=145.79.227.19:4077 , piece_id=VU7DE2EBAGLW7STPZJMH3IPR6FXVOQTXUFWTUH7KSLQIBOVS6CRA, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.097201153 (551h20m26.968284188s) [noise=true , address=145.79.227.6:4259 , piece_id=6OUQWEJO3JCVK4R2CSXHI2USMJMVPWYQSCGYYHUKOA3WOJA2XKCQ, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.097541781 (551h20m26.968624826s) [noise=true , address=145.79.227.37:4566 , piece_id=6RH6NXC5R3U25RAQOHDS42KBJ6QWCS24VRCW6GTKTIIG2FFPHGFQ, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.097932225 (551h20m26.96901526s) [noise=true , address=77.90.31.18:4805 , piece_id=7LO5YQSS2UBZIGEZS7F226PPVMTGLMWKFDDUE2A7WAB3766ILBQA, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.098618392 (551h20m26.969701437s) [noise=true , address=45.91.116.5:4297 , piece_id=75ZWT6KXOSDEE4XG44JHCBU4X5TYYY4FYQ5VHIZSHMGDUBQLRRUQ, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.098971544 (551h20m26.970054579s) [noise=true , address=145.79.227.2:4114 , piece_id=VHSQUG2VEO2ZQZ3DHNLNWYTS4YYHFF35FL7KWWU37GCTAZPLMXUQ, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.099341618 (551h20m26.970424653s) [noise=true , address=45.91.116.8:4048 , piece_id=3PB5EN5FVSYQZ4HUJNXA57A4HRTAQP2GQXECS7MRSFRC7CHXQ7KA, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.099693468 (551h20m26.970776503s) [noise=true , address=145.79.227.25:4619 , piece_id=K56Z6MMBFPBQ5C4ZTZ2JWY6N7FQFEQ6Z7PBDOYL6E5CIFG4MA6LA, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.100159947 (551h20m26.971242992s) [noise=true , address=145.79.227.36:4519 , piece_id=JA3NZNENAPXRYQ65A5RJ4AMOYMIQ75F4KTGH4P6GV3YO4PXXCS4Q, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.100510023 (551h20m26.971593058s) [noise=true , address=145.79.227.50:4373 , piece_id=SGLAGZDBNWO6GTYBAD7257DKCXROXZUDN2RTRFX55AY5UVGTX3RQ, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:28.811420760 (551h20m28.682503805s) [upload=483402]: Next returned nil segment
Dec 28 14:49:28.811513175 (551h20m28.68259621s) [upload=483402]: Waiting for error group managing segments...
Dec 28 14:49:29.101125205 (551h20m28.97220825s) [noise=true , address=145.79.227.25:4619 , piece_id=K56Z6MMBFPBQ5C4ZTZ2JWY6N7FQFEQ6Z7PBDOYL6E5CIFG4MA6LA, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.102240679 (551h20m28.973323724s) [noise=true , address=145.79.227.36:4519 , piece_id=JA3NZNENAPXRYQ65A5RJ4AMOYMIQ75F4KTGH4P6GV3YO4PXXCS4Q, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.103529042 (551h20m28.974612097s) [noise=true , address=145.79.227.19:4077 , piece_id=VU7DE2EBAGLW7STPZJMH3IPR6FXVOQTXUFWTUH7KSLQIBOVS6CRA, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.104590202 (551h20m28.975673247s) [noise=true , address=145.79.227.4:4186 , piece_id=E5RL63ZF2X67454ITCBCP77C3RBY5S5CH4HJQT6EDUAAG4K3MVFQ, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.123887168 (551h20m28.994970203s) [noise=true , address=145.79.227.6:4259 , piece_id=6OUQWEJO3JCVK4R2CSXHI2USMJMVPWYQSCGYYHUKOA3WOJA2XKCQ, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.124475989 (551h20m28.995559054s) [noise=true , address=145.79.227.39:4015 , piece_id=TXO5233BTSNH5F4KXE62VOFKV3A5WEFLJBOM72L5ROU6C4M457XQ, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.124745742 (551h20m28.995828787s) [noise=true , address=145.79.227.37:4566 , piece_id=6RH6NXC5R3U25RAQOHDS42KBJ6QWCS24VRCW6GTKTIIG2FFPHGFQ, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.125303544 (551h20m28.996386579s) [noise=true , address=145.79.227.2:4114 , piece_id=VHSQUG2VEO2ZQZ3DHNLNWYTS4YYHFF35FL7KWWU37GCTAZPLMXUQ, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.128525870 (551h20m28.999608915s) [noise=true , address=145.79.227.50:4373 , piece_id=SGLAGZDBNWO6GTYBAD7257DKCXROXZUDN2RTRFX55AY5UVGTX3RQ, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.130796023 (551h20m29.001879078s) [noise=true , address=145.79.227.42:4123 , piece_id=KBOYHLQK2JYAAAD7U3QFQ6HP54KTQPIB4JZF2RV4H7BHLE3WPCZQ, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.131247102 (551h20m29.002330147s) [noise=true , address=145.79.227.48:4303 , piece_id=YL7JSA5QHJ3GK4NU344BNUIF5TONSV767UTYEE3YMMF4P7BQ35TA, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.313107584 (551h20m29.184190639s) [noise=true , address=45.91.116.6:4976 , piece_id=MJ6RW6NMXBDYGUM2IOBH7HNVFVXSDEJL5UFZKP4FODDP6R6G5OYQ, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.375266024 (551h20m29.246349069s) [noise=true , address=45.91.116.13:4156 , piece_id=SEVVYACKWMXC6I6QQ54JHPNJBTDE3E2V2J7KFL5WTI7QKHGY4YQQ, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.375283747 (551h20m29.246366792s) [seg_pos={0 0}, upload=483402]: Segment reached optimal threshold of 13 pieces.
Dec 28 14:49:29.375478719 (551h20m29.246561754s) [noise=true , address=77.90.31.18:4805 , piece_id=7LO5YQSS2UBZIGEZS7F226PPVMTGLMWKFDDUE2A7WAB3766ILBQA, seg_pos={0 0}, upload=483402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF
Dec 28 14:49:29.375485772 (551h20m29.246568807s) [noise=true , address=77.90.31.18:4805 , piece_id=7LO5YQSS2UBZIGEZS7F226PPVMTGLMWKFDDUE2A7WAB3766ILBQA, seg_pos={0 0}, upload=483402]: Long tail cancellation due to optimal threshold
Dec 28 14:49:29.375489529 (551h20m29.246572564s) [noise=true , address=45.91.116.5:4297 , piece_id=75ZWT6KXOSDEE4XG44JHCBU4X5TYYY4FYQ5VHIZSHMGDUBQLRRUQ, seg_pos={0 0}, upload=483402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF
Dec 28 14:49:29.375498176 (551h20m29.246581231s) [noise=true , address=45.91.116.5:4297 , piece_id=75ZWT6KXOSDEE4XG44JHCBU4X5TYYY4FYQ5VHIZSHMGDUBQLRRUQ, seg_pos={0 0}, upload=483402]: Long tail cancellation due to optimal threshold
Dec 28 14:49:29.375598046 (551h20m29.246681091s) [noise=true , address=45.91.116.8:4048 , piece_id=3PB5EN5FVSYQZ4HUJNXA57A4HRTAQP2GQXECS7MRSFRC7CHXQ7KA, seg_pos={0 0}, upload=483402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF
Dec 28 14:49:29.375603437 (551h20m29.246686472s) [noise=true , address=45.91.116.8:4048 , piece_id=3PB5EN5FVSYQZ4HUJNXA57A4HRTAQP2GQXECS7MRSFRC7CHXQ7KA, seg_pos={0 0}, upload=483402]: Long tail cancellation due to optimal threshold
Dec 28 14:49:29.375617824 (551h20m29.246700859s) [seg_pos={0 0}, upload=483402]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 14:49:29.375687867 (551h20m29.246770912s) [upload=483402]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 14:49:29.375692165 (551h20m29.2467752s) [upload=483402]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 14:49:29.682258328 (551h20m29.553341363s) [upload=483402]: Done uploading segments...
Dec 28 14:49:29.682278987 (551h20m29.553362032s) [upload=483402]: Upload finished. err: <nil>
Duration: 6.004857059s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............X..............................................] duration:
15.978736249s trace:
https://jaeger.dev.storj.io/trace/35a5d4335e2cde85
FAIL (2025-12-28 14:49:58 UTC)
orbiter warning: context deadline exceeded
Duration: 15.978787326s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
7 out of 60 is failed (threshold=3) [.X........X...X............X...............X......X....X....] duration:
32.225574468s trace:
https://jaeger.dev.storj.io/trace/1440a926e26dd799
FAIL (2025-12-28 14:49:57 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
noise=true , address=145.79.227.44:4212 , piece_id=V73IIIU7CCTULBPEXD2YWSFUUGEP2ETVL7LQGIWWEXICZ5SK4BQA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.239195729 (551h16m25.110278774s) [noise=true , address=145.79.227.34:4931 , piece_id=IBNDRK5G5MGWQKRTJ3CS7LI7F3TFUJDHZDG6HJZ44XOQAEWBVEBA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.239549302 (551h16m25.110632347s) [noise=true , address=64.49.13.140:4515 , piece_id=42YNEU3HOVPRPUVOCPKYSQUBTDLBQELQCWNIYJBDXD2RV2NRDJDQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.239966557 (551h16m25.111049602s) [noise=true , address=77.90.31.18:4816 , piece_id=J3DJPEVEESV6EA5KJU6TN7MO4ATQZDBQOSJSH5S5KSZRYGHHZRSQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.240145648 (551h16m25.111228683s) [noise=true , address=77.90.31.14:4688 , piece_id=LNETNIH6PNQ2OR2NQKBWWWOJEFYXIGYHHH34WW7DHTKKYQWJ6DLA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.240933478 (551h16m25.112016523s) [noise=true , address=77.90.31.19:4867 , piece_id=NWQD7PXUGDHUKRF4J3E6EUORCKUM3TDAD2EJH7F3CE55G4EA7SJQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.241389286 (551h16m25.112472331s) [noise=true , address=145.79.227.40:4075 , piece_id=XKJZ2GSNELWKMTMZA5EZ2YTM62XREZQDFI7MDDKAPE3YXRY4KWNA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.241451415 (551h16m25.11253445s) [noise=true , address=77.90.31.21:4889 , piece_id=WSIKD7GCMKHWEFYC7UZ3YWGQHQDIZVW2YJY4BD5ENMCYBJZJMXEQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.241843531 (551h16m25.112926566s) [noise=true , address=145.79.227.46:4608 , piece_id=T5G4T4OU62UPXPIACXJQEZ36O2N63HL4L4HRMW3EN6B4PIE3M7YA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.241911240 (551h16m25.112994285s) [noise=true , address=64.49.13.169:4711 , piece_id=YHY2V4EHCMQEDR6OTZE2POM45Y7FYVNOSEGSUOM5IYWCDWDLRIKQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.242196533 (551h16m25.113279578s) [noise=true , address=193.149.164.143:4697 , piece_id=6SHPKJO6AZLKYPIQWJWNI4YSAMLAVU7BGDGQLQC4I6AGCRWT52DA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.242532343 (551h16m25.113615378s) [noise=true , address=145.79.227.29:4947 , piece_id=FSJZQVSKN76O5RLIVQDPKAN2PDV4VLRCMBRA5EYJGFOETOZ6D7AQ, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.242582478 (551h16m25.113665613s) [noise=true , address=145.79.227.18:4010 , piece_id=GVY67LYRK3UZRX6OLLWC5JCJEGV3RCIS6DATIDMQCBJSL7DCAE4Q, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.242986888 (551h16m25.114069923s) [noise=true , address=64.49.13.27:4966 , piece_id=N3R7JO5PL47V7ZUXNLXFOHSI2MEJ2SVQNHE6M2DWDYF3D4L34ZCA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.243048475 (551h16m25.11413151s) [noise=true , address=77.90.31.11:4585 , piece_id=5MKOQOD3O7TXKXODJWTVZI5EAKBEO7JE3KKXD63F6SO3EBBSVD6A, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:25.243336464 (551h16m25.114419499s) [noise=true , address=77.90.31.17:4801 , piece_id=OWQPIGFIL5SFLUI77IYA3LKVUFPSBL6JSJO6XWZLQFV7GXHF3QOA, seg_pos={0 0}, upload=483348]: Uploading piece...
Dec 28 14:45:26.537548687 (551h16m26.408631722s) [upload=483348]: Next returned nil segment
Dec 28 14:45:26.537559538 (551h16m26.408642573s) [upload=483348]: Waiting for error group managing segments...
Dec 28 14:45:26.899924119 (551h16m26.771007164s) [noise=true , address=64.49.13.140:4515 , piece_id=42YNEU3HOVPRPUVOCPKYSQUBTDLBQELQCWNIYJBDXD2RV2NRDJDQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:26.977957828 (551h16m26.849040873s) [noise=true , address=64.49.13.27:4966 , piece_id=N3R7JO5PL47V7ZUXNLXFOHSI2MEJ2SVQNHE6M2DWDYF3D4L34ZCA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.120995816 (551h16m26.992078862s) [noise=true , address=145.79.227.46:4608 , piece_id=T5G4T4OU62UPXPIACXJQEZ36O2N63HL4L4HRMW3EN6B4PIE3M7YA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.151904071 (551h16m27.022987116s) [noise=true , address=145.79.227.34:4931 , piece_id=IBNDRK5G5MGWQKRTJ3CS7LI7F3TFUJDHZDG6HJZ44XOQAEWBVEBA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.165435146 (551h16m27.036518181s) [noise=true , address=145.79.227.29:4947 , piece_id=FSJZQVSKN76O5RLIVQDPKAN2PDV4VLRCMBRA5EYJGFOETOZ6D7AQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.175130708 (551h16m27.046213754s) [noise=true , address=145.79.227.18:4010 , piece_id=GVY67LYRK3UZRX6OLLWC5JCJEGV3RCIS6DATIDMQCBJSL7DCAE4Q, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.352651587 (551h16m27.223734632s) [noise=true , address=77.90.31.19:4867 , piece_id=NWQD7PXUGDHUKRF4J3E6EUORCKUM3TDAD2EJH7F3CE55G4EA7SJQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.375485928 (551h16m27.246568973s) [noise=true , address=77.90.31.11:4585 , piece_id=5MKOQOD3O7TXKXODJWTVZI5EAKBEO7JE3KKXD63F6SO3EBBSVD6A, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.428947778 (551h16m27.300030823s) [noise=true , address=77.90.31.18:4816 , piece_id=J3DJPEVEESV6EA5KJU6TN7MO4ATQZDBQOSJSH5S5KSZRYGHHZRSQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.567872934 (551h16m27.438955969s) [noise=true , address=77.90.31.17:4801 , piece_id=OWQPIGFIL5SFLUI77IYA3LKVUFPSBL6JSJO6XWZLQFV7GXHF3QOA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:27.983670843 (551h16m27.854753878s) [noise=true , address=77.90.31.13:4650 , piece_id=NHTWFNKCRHELW7BJ2NR4GGHY4LU743RVLCCR236AO4MQUZD7KMEQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:28.019933813 (551h16m27.891016848s) [noise=true , address=145.79.227.30:4794 , piece_id=OZXLKN4V62NRD3WCMF65C7U7OPM2FPBOIMNNE53LN4MBSDDL65NQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:28.083140990 (551h16m27.954224025s) [noise=true , address=145.79.227.44:4212 , piece_id=V73IIIU7CCTULBPEXD2YWSFUUGEP2ETVL7LQGIWWEXICZ5SK4BQA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:28.129315084 (551h16m28.000398119s) [noise=true , address=145.79.227.40:4075 , piece_id=XKJZ2GSNELWKMTMZA5EZ2YTM62XREZQDFI7MDDKAPE3YXRY4KWNA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: <nil>
Dec 28 14:45:28.129340211 (551h16m28.000423246s) [seg_pos={0 0}, upload=483348]: Segment reached optimal threshold of 14 pieces.
Dec 28 14:45:28.129532938 (551h16m28.000615973s) [noise=true , address=77.90.31.21:4889 , piece_id=WSIKD7GCMKHWEFYC7UZ3YWGQHQDIZVW2YJY4BD5ENMCYBJZJMXEQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wcX8iEhi5LrftcXUQE3Lrm4igNwhDtvT3RNBERTGsKyQ6KLj6): protocol: expected piece hash; context canceled; EOF
Dec 28 14:45:28.129539380 (551h16m28.000622425s) [noise=true , address=77.90.31.21:4889 , piece_id=WSIKD7GCMKHWEFYC7UZ3YWGQHQDIZVW2YJY4BD5ENMCYBJZJMXEQ, seg_pos={0 0}, upload=483348]: Long tail cancellation due to optimal threshold
Dec 28 14:45:28.129666382 (551h16m28.000749418s) [noise=true , address=64.49.13.169:4711 , piece_id=YHY2V4EHCMQEDR6OTZE2POM45Y7FYVNOSEGSUOM5IYWCDWDLRIKQ, seg_pos={0 0}, upload=483348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:47258->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:47258->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47258->64.49.13.169:4711: use of closed network connection
Dec 28 14:45:28.129671853 (551h16m28.000754898s) [noise=true , address=64.49.13.169:4711 , piece_id=YHY2V4EHCMQEDR6OTZE2POM45Y7FYVNOSEGSUOM5IYWCDWDLRIKQ, seg_pos={0 0}, upload=483348]: Long tail cancellation due to optimal threshold
Dec 28 14:45:28.129713071 (551h16m28.000796116s) [noise=true , address=77.90.31.14:4688 , piece_id=LNETNIH6PNQ2OR2NQKBWWWOJEFYXIGYHHH34WW7DHTKKYQWJ6DLA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz): protocol: expected piece hash; context canceled; EOF
Dec 28 14:45:28.129770090 (551h16m28.000853125s) [noise=true , address=77.90.31.14:4688 , piece_id=LNETNIH6PNQ2OR2NQKBWWWOJEFYXIGYHHH34WW7DHTKKYQWJ6DLA, seg_pos={0 0}, upload=483348]: Long tail cancellation due to optimal threshold
Dec 28 14:45:28.129862666 (551h16m28.000945711s) [noise=true , address=193.149.164.143:4697 , piece_id=6SHPKJO6AZLKYPIQWJWNI4YSAMLAVU7BGDGQLQC4I6AGCRWT52DA, seg_pos={0 0}, upload=483348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ): protocol: expected piece hash; context canceled; EOF
Dec 28 14:45:28.129870651 (551h16m28.000953686s) [noise=true , address=193.149.164.143:4697 , piece_id=6SHPKJO6AZLKYPIQWJWNI4YSAMLAVU7BGDGQLQC4I6AGCRWT52DA, seg_pos={0 0}, upload=483348]: Long tail cancellation due to optimal threshold
Dec 28 14:45:28.129970341 (551h16m28.001053376s) [seg_pos={0 0}, upload=483348]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 14:45:28.130053620 (551h16m28.001136665s) [upload=483348]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 14:45:28.130060553 (551h16m28.001143588s) [upload=483348]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 14:45:28.417819022 (551h16m28.288902057s) [upload=483348]: Done uploading segments...
Dec 28 14:45:28.417844872 (551h16m28.288927927s) [upload=483348]: Upload finished. err: <nil>
Duration: 32.225602612s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
21 out of 60 is failed (threshold=3) [.XX......XX.XX.X.X..XXXXX....X...X..X..X...X...X.X.........X] duration:
30.003214898s trace:
https://jaeger.dev.storj.io/trace/550d1e289028a496
FAIL (2025-12-28 14:49:56 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 28 14:49:26.847085842 (551h20m26.718168877s) [upload=483402]: Starting upload
Dec 28 14:49:26.847246297 (551h20m26.718329332s) [upload=483402]: Done starting upload
Dec 28 14:49:26.847480553 (551h20m26.718563588s) [upload=483402]: Uploading segments...
Dec 28 14:49:26.847534546 (551h20m26.718617581s) [upload=483402]: Got next segment. Inline: false
Dec 28 14:49:26.847611382 (551h20m26.718694418s) [upload=483402]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 14:49:26.847625389 (551h20m26.718708424s) [upload=483402]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 14:49:27.094997497 (551h20m26.966080542s) [seg_pos={0 0}, upload=483402]: Begin upload segment...
Dec 28 14:49:27.095052100 (551h20m26.966135145s) [seg_pos={0 0}, upload=483402]: Done begin upload segment.
Dec 28 14:49:27.095090593 (551h20m26.966173638s) [noise=true , address=45.91.116.13:4156 , piece_id=SEVVYACKWMXC6I6QQ54JHPNJBTDE3E2V2J7KFL5WTI7QKHGY4YQQ, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.095108959 (551h20m26.966191994s) [noise=true , address=145.79.227.48:4303 , piece_id=YL7JSA5QHJ3GK4NU344BNUIF5TONSV767UTYEE3YMMF4P7BQ35TA, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.095613940 (551h20m26.966696975s) [noise=true , address=145.79.227.39:4015 , piece_id=TXO5233BTSNH5F4KXE62VOFKV3A5WEFLJBOM72L5ROU6C4M457XQ, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.095909944 (551h20m26.966992979s) [noise=true , address=145.79.227.4:4186 , piece_id=E5RL63ZF2X67454ITCBCP77C3RBY5S5CH4HJQT6EDUAAG4K3MVFQ, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.096198974 (551h20m26.967282009s) [noise=true , address=145.79.227.42:4123 , piece_id=KBOYHLQK2JYAAAD7U3QFQ6HP54KTQPIB4JZF2RV4H7BHLE3WPCZQ, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.096496671 (551h20m26.967579706s) [noise=true , address=45.91.116.6:4976 , piece_id=MJ6RW6NMXBDYGUM2IOBH7HNVFVXSDEJL5UFZKP4FODDP6R6G5OYQ, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.096817342 (551h20m26.967900377s) [noise=true , address=145.79.227.19:4077 , piece_id=VU7DE2EBAGLW7STPZJMH3IPR6FXVOQTXUFWTUH7KSLQIBOVS6CRA, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.097201153 (551h20m26.968284188s) [noise=true , address=145.79.227.6:4259 , piece_id=6OUQWEJO3JCVK4R2CSXHI2USMJMVPWYQSCGYYHUKOA3WOJA2XKCQ, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.097541781 (551h20m26.968624826s) [noise=true , address=145.79.227.37:4566 , piece_id=6RH6NXC5R3U25RAQOHDS42KBJ6QWCS24VRCW6GTKTIIG2FFPHGFQ, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.097932225 (551h20m26.96901526s) [noise=true , address=77.90.31.18:4805 , piece_id=7LO5YQSS2UBZIGEZS7F226PPVMTGLMWKFDDUE2A7WAB3766ILBQA, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.098618392 (551h20m26.969701437s) [noise=true , address=45.91.116.5:4297 , piece_id=75ZWT6KXOSDEE4XG44JHCBU4X5TYYY4FYQ5VHIZSHMGDUBQLRRUQ, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.098971544 (551h20m26.970054579s) [noise=true , address=145.79.227.2:4114 , piece_id=VHSQUG2VEO2ZQZ3DHNLNWYTS4YYHFF35FL7KWWU37GCTAZPLMXUQ, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.099341618 (551h20m26.970424653s) [noise=true , address=45.91.116.8:4048 , piece_id=3PB5EN5FVSYQZ4HUJNXA57A4HRTAQP2GQXECS7MRSFRC7CHXQ7KA, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.099693468 (551h20m26.970776503s) [noise=true , address=145.79.227.25:4619 , piece_id=K56Z6MMBFPBQ5C4ZTZ2JWY6N7FQFEQ6Z7PBDOYL6E5CIFG4MA6LA, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.100159947 (551h20m26.971242992s) [noise=true , address=145.79.227.36:4519 , piece_id=JA3NZNENAPXRYQ65A5RJ4AMOYMIQ75F4KTGH4P6GV3YO4PXXCS4Q, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:27.100510023 (551h20m26.971593058s) [noise=true , address=145.79.227.50:4373 , piece_id=SGLAGZDBNWO6GTYBAD7257DKCXROXZUDN2RTRFX55AY5UVGTX3RQ, seg_pos={0 0}, upload=483402]: Uploading piece...
Dec 28 14:49:28.811420760 (551h20m28.682503805s) [upload=483402]: Next returned nil segment
Dec 28 14:49:28.811513175 (551h20m28.68259621s) [upload=483402]: Waiting for error group managing segments...
Dec 28 14:49:29.101125205 (551h20m28.97220825s) [noise=true , address=145.79.227.25:4619 , piece_id=K56Z6MMBFPBQ5C4ZTZ2JWY6N7FQFEQ6Z7PBDOYL6E5CIFG4MA6LA, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.102240679 (551h20m28.973323724s) [noise=true , address=145.79.227.36:4519 , piece_id=JA3NZNENAPXRYQ65A5RJ4AMOYMIQ75F4KTGH4P6GV3YO4PXXCS4Q, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.103529042 (551h20m28.974612097s) [noise=true , address=145.79.227.19:4077 , piece_id=VU7DE2EBAGLW7STPZJMH3IPR6FXVOQTXUFWTUH7KSLQIBOVS6CRA, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.104590202 (551h20m28.975673247s) [noise=true , address=145.79.227.4:4186 , piece_id=E5RL63ZF2X67454ITCBCP77C3RBY5S5CH4HJQT6EDUAAG4K3MVFQ, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.123887168 (551h20m28.994970203s) [noise=true , address=145.79.227.6:4259 , piece_id=6OUQWEJO3JCVK4R2CSXHI2USMJMVPWYQSCGYYHUKOA3WOJA2XKCQ, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.124475989 (551h20m28.995559054s) [noise=true , address=145.79.227.39:4015 , piece_id=TXO5233BTSNH5F4KXE62VOFKV3A5WEFLJBOM72L5ROU6C4M457XQ, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.124745742 (551h20m28.995828787s) [noise=true , address=145.79.227.37:4566 , piece_id=6RH6NXC5R3U25RAQOHDS42KBJ6QWCS24VRCW6GTKTIIG2FFPHGFQ, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.125303544 (551h20m28.996386579s) [noise=true , address=145.79.227.2:4114 , piece_id=VHSQUG2VEO2ZQZ3DHNLNWYTS4YYHFF35FL7KWWU37GCTAZPLMXUQ, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.128525870 (551h20m28.999608915s) [noise=true , address=145.79.227.50:4373 , piece_id=SGLAGZDBNWO6GTYBAD7257DKCXROXZUDN2RTRFX55AY5UVGTX3RQ, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.130796023 (551h20m29.001879078s) [noise=true , address=145.79.227.42:4123 , piece_id=KBOYHLQK2JYAAAD7U3QFQ6HP54KTQPIB4JZF2RV4H7BHLE3WPCZQ, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.131247102 (551h20m29.002330147s) [noise=true , address=145.79.227.48:4303 , piece_id=YL7JSA5QHJ3GK4NU344BNUIF5TONSV767UTYEE3YMMF4P7BQ35TA, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.313107584 (551h20m29.184190639s) [noise=true , address=45.91.116.6:4976 , piece_id=MJ6RW6NMXBDYGUM2IOBH7HNVFVXSDEJL5UFZKP4FODDP6R6G5OYQ, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.375266024 (551h20m29.246349069s) [noise=true , address=45.91.116.13:4156 , piece_id=SEVVYACKWMXC6I6QQ54JHPNJBTDE3E2V2J7KFL5WTI7QKHGY4YQQ, seg_pos={0 0}, upload=483402]: Done uploading piece. err: <nil>
Dec 28 14:49:29.375283747 (551h20m29.246366792s) [seg_pos={0 0}, upload=483402]: Segment reached optimal threshold of 13 pieces.
Dec 28 14:49:29.375478719 (551h20m29.246561754s) [noise=true , address=77.90.31.18:4805 , piece_id=7LO5YQSS2UBZIGEZS7F226PPVMTGLMWKFDDUE2A7WAB3766ILBQA, seg_pos={0 0}, upload=483402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF
Dec 28 14:49:29.375485772 (551h20m29.246568807s) [noise=true , address=77.90.31.18:4805 , piece_id=7LO5YQSS2UBZIGEZS7F226PPVMTGLMWKFDDUE2A7WAB3766ILBQA, seg_pos={0 0}, upload=483402]: Long tail cancellation due to optimal threshold
Dec 28 14:49:29.375489529 (551h20m29.246572564s) [noise=true , address=45.91.116.5:4297 , piece_id=75ZWT6KXOSDEE4XG44JHCBU4X5TYYY4FYQ5VHIZSHMGDUBQLRRUQ, seg_pos={0 0}, upload=483402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF
Dec 28 14:49:29.375498176 (551h20m29.246581231s) [noise=true , address=45.91.116.5:4297 , piece_id=75ZWT6KXOSDEE4XG44JHCBU4X5TYYY4FYQ5VHIZSHMGDUBQLRRUQ, seg_pos={0 0}, upload=483402]: Long tail cancellation due to optimal threshold
Dec 28 14:49:29.375598046 (551h20m29.246681091s) [noise=true , address=45.91.116.8:4048 , piece_id=3PB5EN5FVSYQZ4HUJNXA57A4HRTAQP2GQXECS7MRSFRC7CHXQ7KA, seg_pos={0 0}, upload=483402]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF
Dec 28 14:49:29.375603437 (551h20m29.246686472s) [noise=true , address=45.91.116.8:4048 , piece_id=3PB5EN5FVSYQZ4HUJNXA57A4HRTAQP2GQXECS7MRSFRC7CHXQ7KA, seg_pos={0 0}, upload=483402]: Long tail cancellation due to optimal threshold
Dec 28 14:49:29.375617824 (551h20m29.246700859s) [seg_pos={0 0}, upload=483402]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 14:49:29.375687867 (551h20m29.246770912s) [upload=483402]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 14:49:29.375692165 (551h20m29.2467752s) [upload=483402]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 14:49:29.682258328 (551h20m29.553341363s) [upload=483402]: Done uploading segments...
Dec 28 14:49:29.682278987 (551h20m29.553362032s) [upload=483402]: Upload finished. err: <nil>
Duration: 30.0033468s
core
uplink
us-select-1
|