Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 7.697813062s trace: https://jaeger.dev.storj.io/trace/2b03c00e8c9bd6a8 FAIL (2025-11-22 12:28:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.697830565s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 8.052062539s trace: https://jaeger.dev.storj.io/trace/666b0e258bc1797c FAIL (2025-11-22 12:28:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.052084821s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.24189208s trace: https://jaeger.dev.storj.io/trace/78d837dffbcbc1eb FAIL (2025-11-22 12:28:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== Nov 22 12:27:49.568493454 (316h23m48.119963905s) [noise=true , address=77.90.31.19:4876 , piece_id=RDATPSCRLI5BNZVPSBOSAPUK5GFYD2MWCIPRAL6NMPA4Z6GDNZ5A, seg_pos={0 0}, upload=278652]: Uploading piece... Nov 22 12:27:49.570659288 (316h23m48.122129749s) [noise=true , address=45.91.116.5:4306 , piece_id=F437446OQSONT5YJK35AQ26QMDMC3HP2CWUPEP46WKZGWNRL2V5A, seg_pos={0 0}, upload=278652]: Uploading piece... Nov 22 12:27:49.572948898 (316h23m48.124419359s) [noise=true , address=64.49.13.140:4506 , piece_id=B5MGJGEBUK6H2JD6LVVKYO34GYGTKQ2RRLTT53QMJHDJGLAH2PPA, seg_pos={0 0}, upload=278652]: Uploading piece... Nov 22 12:27:49.573111158 (316h23m48.124581609s) [noise=true , address=45.91.116.4:4948 , piece_id=WNZONHIDT5WLFQW6ZYMZOOQ2DT4ATWKBMRGKVLI5SKY53MAMCEYQ, seg_pos={0 0}, upload=278652]: Uploading piece... Nov 22 12:27:49.574907569 (316h23m48.12637803s) [noise=true , address=45.91.116.10:4087 , piece_id=W4QTE4CCAQGK6KODK2DFBY6YEGEGYOHZS7WBJMV7IN3PCHOMOG6A, seg_pos={0 0}, upload=278652]: Uploading piece... Nov 22 12:27:49.576925240 (316h23m48.128395701s) [noise=true , address=77.90.31.10:4539 , piece_id=I5AHSBYZXFUVBLW4L4R4CFVJOFFPLQB7F5AK2GBYDXISM6N72PDQ, seg_pos={0 0}, upload=278652]: Uploading piece... Nov 22 12:27:49.579508780 (316h23m48.130979241s) [noise=true , address=145.79.227.33:4894 , piece_id=QD6HDQDS37Q6HOIU522KZZ44LXF2AKRVKL5PIQFG437KTXYMKC6Q, seg_pos={0 0}, upload=278652]: Uploading piece... Nov 22 12:27:49.580586642 (316h23m48.132057104s) [noise=true , address=77.90.31.13:4630 , piece_id=FNU2RL7WP5G4RR3HMDEEKYKNJBLTH6SKZ3FFPZK4B7U4A2RRWWKQ, seg_pos={0 0}, upload=278652]: Uploading piece... Nov 22 12:27:49.585268507 (316h23m48.136738968s) [noise=true , address=77.90.31.12:4608 , piece_id=X2N6DK2U6TUZXCP356BDGEBOWY44UEOCG73CWEY57GY7OA7ZTNXA, seg_pos={0 0}, upload=278652]: Uploading piece... Nov 22 12:27:49.588099719 (316h23m48.13957018s) [noise=true , address=45.91.116.6:4986 , piece_id=RHKQEB2Y7KCD4CIROR6DSYPTVG2SUNDKE7C5QW7DHHM24YJHMNOQ, seg_pos={0 0}, upload=278652]: Uploading piece... Nov 22 12:27:49.589734080 (316h23m48.141204531s) [noise=true , address=145.79.227.12:4790 , piece_id=ZCJGR7J7ZDQWNMWSLFPHSGRGRQYLNMDSO52Z2TFQCQCBXIBVPZYA, seg_pos={0 0}, upload=278652]: Uploading piece... Nov 22 12:27:49.592364720 (316h23m48.143835201s) [noise=true , address=77.90.31.17:4799 , piece_id=K2U42DVPQMOBOXQ4J2FZDWEQURTV42XSVR23AT32YCKEXAPD24RA, seg_pos={0 0}, upload=278652]: Uploading piece... Nov 22 12:27:49.600851380 (316h23m48.152321841s) [noise=true , address=145.79.227.7:4314 , piece_id=D2CQYNQ7U5IP4QRSTASQIO5JBXYPWSHOP3HN4SVVOTN6LIQNSN3A, seg_pos={0 0}, upload=278652]: Uploading piece... Nov 22 12:27:50.972898701 (316h23m49.524369162s) [upload=278652]: Next returned nil segment Nov 22 12:27:50.972914963 (316h23m49.524385414s) [upload=278652]: Waiting for error group managing segments... Nov 22 12:27:51.338580227 (316h23m49.890050688s) [noise=true , address=64.49.13.140:4506 , piece_id=B5MGJGEBUK6H2JD6LVVKYO34GYGTKQ2RRLTT53QMJHDJGLAH2PPA, seg_pos={0 0}, upload=278652]: Done uploading piece. err: <nil> Nov 22 12:27:51.425396702 (316h23m49.976867163s) [noise=true , address=145.79.227.5:4256 , piece_id=UOFRVFML2YA53NWWQZMB3GA5PW4NFULCJJHMALH7CHHNGSG3N7VA, seg_pos={0 0}, upload=278652]: Done uploading piece. err: <nil> Nov 22 12:27:51.444596362 (316h23m49.996066823s) [noise=true , address=145.79.227.49:4354 , piece_id=357B2N2MEKIVI7ERX5GY33FDAZOO2A3ENZV74UVPEBYSYDNI5GIQ, seg_pos={0 0}, upload=278652]: Done uploading piece. err: <nil> Nov 22 12:27:51.448630013 (316h23m50.000100474s) [noise=true , address=145.79.227.33:4894 , piece_id=QD6HDQDS37Q6HOIU522KZZ44LXF2AKRVKL5PIQFG437KTXYMKC6Q, seg_pos={0 0}, upload=278652]: Done uploading piece. err: <nil> Nov 22 12:27:51.466112773 (316h23m50.017583234s) [noise=true , address=145.79.227.12:4790 , piece_id=ZCJGR7J7ZDQWNMWSLFPHSGRGRQYLNMDSO52Z2TFQCQCBXIBVPZYA, seg_pos={0 0}, upload=278652]: Done uploading piece. err: <nil> Nov 22 12:27:51.471437682 (316h23m50.022908133s) [noise=true , address=145.79.227.7:4314 , piece_id=D2CQYNQ7U5IP4QRSTASQIO5JBXYPWSHOP3HN4SVVOTN6LIQNSN3A, seg_pos={0 0}, upload=278652]: Done uploading piece. err: <nil> Nov 22 12:28:48.617770024 (316h24m47.169240485s) [noise=true , address=77.90.31.17:4799 , piece_id=K2U42DVPQMOBOXQ4J2FZDWEQURTV42XSVR23AT32YCKEXAPD24RA, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:12UhAksgXZuFGAVheaYzP7D4kwVyAjUAq3BtrvdSxszjNCFCTWT, address:77.90.31.17:4799): protocol: rpc: dial tcp 77.90.31.17:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:28:48.617979923 (316h24m47.169450384s) [noise=true , address=45.91.116.5:4306 , piece_id=F437446OQSONT5YJK35AQ26QMDMC3HP2CWUPEP46WKZGWNRL2V5A, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:1QuFvRW9wGKdKRXPK8UdR8uqTm53CnHaF9CsUz2bUh2Ptw9r1F, address:45.91.116.5:4306): protocol: rpc: dial tcp 45.91.116.5:4306: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:28:48.618033846 (316h24m47.169504308s) [noise=true , address=77.90.31.21:4885 , piece_id=5OL3NVLKPFB23XQFUDQLBX6EJHPF5VICQYSYQKCSDY5SWHILW4QA, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:12TfMGrvCMPvQ9ZC1pVoDqjFnUHXtzwwnZB6pCV1em4pdmbK2MP, address:77.90.31.21:4885): protocol: rpc: dial tcp 77.90.31.21:4885: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:28:48.618228647 (316h24m47.169699098s) [noise=true , address=45.91.116.4:4948 , piece_id=WNZONHIDT5WLFQW6ZYMZOOQ2DT4ATWKBMRGKVLI5SKY53MAMCEYQ, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:184g6qHdezLW3d2CKNW8GRgdzeVLQuMEhd8B8c1zEnkKNkeSTu, address:45.91.116.4:4948): protocol: rpc: dial tcp 45.91.116.4:4948: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:28:48.618249357 (316h24m47.169719848s) [noise=true , address=45.91.116.9:4279 , piece_id=LSPEKODU2YPLRUI6MY37VVRL6QNUJ2IXTZX6BBJHEVDMLTZGNQVQ, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:124aq2BHBfu1qSRXCx5XVZJU5U7gzadYHSRLYT8fcwcrtu9z5jG, address:45.91.116.9:4279): protocol: rpc: dial tcp 45.91.116.9:4279: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:28:48.618444829 (316h24m47.16991529s) [noise=true , address=77.90.31.10:4539 , piece_id=I5AHSBYZXFUVBLW4L4R4CFVJOFFPLQB7F5AK2GBYDXISM6N72PDQ, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka, address:77.90.31.10:4539): protocol: rpc: dial tcp 77.90.31.10:4539: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:28:48.618487821 (316h24m47.169958272s) [noise=true , address=45.91.116.14:4211 , piece_id=5HS3X7L6ACA6KJNWBBEK36IIBOFNANCMF4QL367VJA5OHCJW4SOQ, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:1TkBbeKqsAZHyZdNpDgCb2uexqgV59Atgtv7EM1rd4KJEZ4BqF, address:45.91.116.14:4211): protocol: rpc: dial tcp 45.91.116.14:4211: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:28:48.618612950 (316h24m47.170083451s) [noise=true , address=45.91.116.6:4986 , piece_id=RHKQEB2Y7KCD4CIROR6DSYPTVG2SUNDKE7C5QW7DHHM24YJHMNOQ, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: write tcp 188.166.225.170:41196->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:28:48.618698352 (316h24m47.170168813s) [noise=true , address=77.90.31.13:4630 , piece_id=FNU2RL7WP5G4RR3HMDEEKYKNJBLTH6SKZ3FFPZK4B7U4A2RRWWKQ, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn, address:77.90.31.13:4630): protocol: rpc: dial tcp 77.90.31.13:4630: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:28:48.618875479 (316h24m47.17034594s) [noise=true , address=77.90.31.12:4608 , piece_id=X2N6DK2U6TUZXCP356BDGEBOWY44UEOCG73CWEY57GY7OA7ZTNXA, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu, address:77.90.31.12:4608): protocol: rpc: dial tcp 77.90.31.12:4608: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:28:48.619011127 (316h24m47.170481588s) [noise=true , address=77.90.31.19:4876 , piece_id=RDATPSCRLI5BNZVPSBOSAPUK5GFYD2MWCIPRAL6NMPA4Z6GDNZ5A, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:12H3UNBGeAufYSdVvEgYkv2QWeFtjKMvHvYGeqGeBvjEHqDDjBB, address:77.90.31.19:4876): protocol: rpc: dial tcp 77.90.31.19:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:28:48.619102862 (316h24m47.170573323s) [noise=true , address=45.91.116.10:4087 , piece_id=W4QTE4CCAQGK6KODK2DFBY6YEGEGYOHZS7WBJMV7IN3PCHOMOG6A, seg_pos={0 0}, upload=278652]: Done uploading piece. err: ecclient: upload failed (node:125T5WzC4RDcp21oJ8zBMMBq5uw2X32Wf5FszZiWUdsSR95Pd2H, address:45.91.116.10:4087): protocol: write tcp 188.166.225.170:50656->45.91.116.10:4087: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 12:28:48.619135034 (316h24m47.170605495s) [seg_pos={0 0}, upload=278652]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Nov 22 12:28:48.858048038 (316h24m47.409518489s) [upload=278652]: Done uploading segments... Nov 22 12:28:48.858131627 (316h24m47.409602088s) [upload=278652]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 1m0.241905786s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.001069757s trace: https://jaeger.dev.storj.io/trace/2b03c00e8c9bd6a8 FAIL (2025-11-22 12:28:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001096567s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.001521449s trace: https://jaeger.dev.storj.io/trace/666b0e258bc1797c FAIL (2025-11-22 12:28:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001573819s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...........X...............................................X] duration: 5.001156266s trace: https://jaeger.dev.storj.io/trace/69bcc2ce5cc82585 FAIL (2025-11-22 12:28:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001188329s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.857497921s trace: https://jaeger.dev.storj.io/trace/76f1ecbff69b5d4e FAIL (2025-11-22 12:27:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, Piece ID: I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ) context deadline exceeded; piecestore: (Node ID: 12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, Piece ID: YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA) context deadline exceeded; piecestore: (Node ID: 1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, Piece ID: G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ) context deadline exceeded; piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA) context deadline exceeded ======LOGS====== nts... Nov 22 11:53:53.345386809 (315h49m51.89685727s) [upload=278145]: Got next segment. Inline: false Nov 22 11:53:53.345428137 (315h49m51.896898598s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 11:53:53.345433948 (315h49m51.896904419s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 11:53:54.603541081 (315h49m53.155011552s) [seg_pos={0 0}, upload=278145]: Begin upload segment... Nov 22 11:53:54.603595445 (315h49m53.155065916s) [seg_pos={0 0}, upload=278145]: Done begin upload segment. Nov 22 11:53:54.603668043 (315h49m53.155138494s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.603684554 (315h49m53.155155005s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604505828 (315h49m53.155976289s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604973038 (315h49m53.156443499s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.605425900 (315h49m53.156896361s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606323339 (315h49m53.15779381s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606878326 (315h49m53.158348777s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.607588267 (315h49m53.159058728s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610211763 (315h49m53.161682215s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610336751 (315h49m53.161807212s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612196994 (315h49m53.163667455s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612485593 (315h49m53.163956064s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614299567 (315h49m53.165770028s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614454202 (315h49m53.165924673s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.616397682 (315h49m53.167868143s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.621720077 (315h49m53.173190538s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:56.445932556 (315h49m54.997403007s) [upload=278145]: Next returned nil segment Nov 22 11:53:56.445949048 (315h49m54.997419509s) [upload=278145]: Waiting for error group managing segments... Nov 22 11:53:56.734588580 (315h49m55.286059041s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.796169791 (315h49m55.347640252s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.800552848 (315h49m55.352023309s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.802134119 (315h49m55.35360458s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.811866140 (315h49m55.363336591s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.820962632 (315h49m55.372433093s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.853755886 (315h49m55.405226347s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.857215724 (315h49m55.408686185s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.067648917 (315h49m55.619119378s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.093474313 (315h49m55.644944774s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.104509907 (315h49m55.655980378s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.505088550 (315h49m56.056559011s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609615419 (315h49m56.16108589s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609676946 (315h49m56.161147437s) [seg_pos={0 0}, upload=278145]: Segment reached optimal threshold of 13 pieces. Nov 22 11:53:57.609972077 (315h49m56.161442539s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection Nov 22 11:53:57.609982889 (315h49m56.16145336s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610025941 (315h49m56.161496432s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 22 11:53:57.610037101 (315h49m56.161507553s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610213978 (315h49m56.161684439s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection Nov 22 11:53:57.610224649 (315h49m56.16169511s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610261058 (315h49m56.16173152s) [seg_pos={0 0}, upload=278145]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 11:53:57.610421113 (315h49m56.161891584s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 11:53:57.610430852 (315h49m56.161901313s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 11:54:00.212174190 (315h49m58.763644651s) [upload=278145]: Done uploading segments... Nov 22 11:54:00.212200729 (315h49m58.763671191s) [upload=278145]: Upload finished. err: <nil> Duration: 5.857525733s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............X...............................................] duration: 4.526657001s trace: https://jaeger.dev.storj.io/trace/469b06ffc0b434d4 FAIL (2025-11-22 12:27:10 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.526685354s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.757537308s trace: https://jaeger.dev.storj.io/trace/76f1ecbff69b5d4e FAIL (2025-11-22 12:27:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, Piece ID: I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ) context deadline exceeded; piecestore: (Node ID: 12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, Piece ID: YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA) context deadline exceeded; piecestore: (Node ID: 1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, Piece ID: G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ) context deadline exceeded; piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA) context deadline exceeded ======LOGS====== nts... Nov 22 11:53:53.345386809 (315h49m51.89685727s) [upload=278145]: Got next segment. Inline: false Nov 22 11:53:53.345428137 (315h49m51.896898598s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 11:53:53.345433948 (315h49m51.896904419s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 11:53:54.603541081 (315h49m53.155011552s) [seg_pos={0 0}, upload=278145]: Begin upload segment... Nov 22 11:53:54.603595445 (315h49m53.155065916s) [seg_pos={0 0}, upload=278145]: Done begin upload segment. Nov 22 11:53:54.603668043 (315h49m53.155138494s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.603684554 (315h49m53.155155005s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604505828 (315h49m53.155976289s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604973038 (315h49m53.156443499s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.605425900 (315h49m53.156896361s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606323339 (315h49m53.15779381s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606878326 (315h49m53.158348777s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.607588267 (315h49m53.159058728s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610211763 (315h49m53.161682215s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610336751 (315h49m53.161807212s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612196994 (315h49m53.163667455s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612485593 (315h49m53.163956064s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614299567 (315h49m53.165770028s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614454202 (315h49m53.165924673s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.616397682 (315h49m53.167868143s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.621720077 (315h49m53.173190538s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:56.445932556 (315h49m54.997403007s) [upload=278145]: Next returned nil segment Nov 22 11:53:56.445949048 (315h49m54.997419509s) [upload=278145]: Waiting for error group managing segments... Nov 22 11:53:56.734588580 (315h49m55.286059041s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.796169791 (315h49m55.347640252s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.800552848 (315h49m55.352023309s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.802134119 (315h49m55.35360458s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.811866140 (315h49m55.363336591s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.820962632 (315h49m55.372433093s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.853755886 (315h49m55.405226347s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.857215724 (315h49m55.408686185s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.067648917 (315h49m55.619119378s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.093474313 (315h49m55.644944774s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.104509907 (315h49m55.655980378s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.505088550 (315h49m56.056559011s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609615419 (315h49m56.16108589s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609676946 (315h49m56.161147437s) [seg_pos={0 0}, upload=278145]: Segment reached optimal threshold of 13 pieces. Nov 22 11:53:57.609972077 (315h49m56.161442539s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection Nov 22 11:53:57.609982889 (315h49m56.16145336s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610025941 (315h49m56.161496432s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 22 11:53:57.610037101 (315h49m56.161507553s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610213978 (315h49m56.161684439s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection Nov 22 11:53:57.610224649 (315h49m56.16169511s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610261058 (315h49m56.16173152s) [seg_pos={0 0}, upload=278145]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 11:53:57.610421113 (315h49m56.161891584s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 11:53:57.610430852 (315h49m56.161901313s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 11:54:00.212174190 (315h49m58.763644651s) [upload=278145]: Done uploading segments... Nov 22 11:54:00.212200729 (315h49m58.763671191s) [upload=278145]: Upload finished. err: <nil> Duration: 5.757566745s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.828794977s trace: https://jaeger.dev.storj.io/trace/76f1ecbff69b5d4e FAIL (2025-11-22 12:26:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, Piece ID: I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ) context deadline exceeded; piecestore: (Node ID: 12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, Piece ID: YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA) context deadline exceeded; piecestore: (Node ID: 1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, Piece ID: G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ) context deadline exceeded; piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA) context deadline exceeded ======LOGS====== nts... Nov 22 11:53:53.345386809 (315h49m51.89685727s) [upload=278145]: Got next segment. Inline: false Nov 22 11:53:53.345428137 (315h49m51.896898598s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 11:53:53.345433948 (315h49m51.896904419s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 11:53:54.603541081 (315h49m53.155011552s) [seg_pos={0 0}, upload=278145]: Begin upload segment... Nov 22 11:53:54.603595445 (315h49m53.155065916s) [seg_pos={0 0}, upload=278145]: Done begin upload segment. Nov 22 11:53:54.603668043 (315h49m53.155138494s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.603684554 (315h49m53.155155005s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604505828 (315h49m53.155976289s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604973038 (315h49m53.156443499s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.605425900 (315h49m53.156896361s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606323339 (315h49m53.15779381s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606878326 (315h49m53.158348777s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.607588267 (315h49m53.159058728s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610211763 (315h49m53.161682215s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610336751 (315h49m53.161807212s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612196994 (315h49m53.163667455s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612485593 (315h49m53.163956064s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614299567 (315h49m53.165770028s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614454202 (315h49m53.165924673s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.616397682 (315h49m53.167868143s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.621720077 (315h49m53.173190538s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:56.445932556 (315h49m54.997403007s) [upload=278145]: Next returned nil segment Nov 22 11:53:56.445949048 (315h49m54.997419509s) [upload=278145]: Waiting for error group managing segments... Nov 22 11:53:56.734588580 (315h49m55.286059041s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.796169791 (315h49m55.347640252s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.800552848 (315h49m55.352023309s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.802134119 (315h49m55.35360458s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.811866140 (315h49m55.363336591s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.820962632 (315h49m55.372433093s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.853755886 (315h49m55.405226347s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.857215724 (315h49m55.408686185s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.067648917 (315h49m55.619119378s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.093474313 (315h49m55.644944774s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.104509907 (315h49m55.655980378s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.505088550 (315h49m56.056559011s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609615419 (315h49m56.16108589s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609676946 (315h49m56.161147437s) [seg_pos={0 0}, upload=278145]: Segment reached optimal threshold of 13 pieces. Nov 22 11:53:57.609972077 (315h49m56.161442539s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection Nov 22 11:53:57.609982889 (315h49m56.16145336s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610025941 (315h49m56.161496432s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 22 11:53:57.610037101 (315h49m56.161507553s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610213978 (315h49m56.161684439s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection Nov 22 11:53:57.610224649 (315h49m56.16169511s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610261058 (315h49m56.16173152s) [seg_pos={0 0}, upload=278145]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 11:53:57.610421113 (315h49m56.161891584s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 11:53:57.610430852 (315h49m56.161901313s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 11:54:00.212174190 (315h49m58.763644651s) [upload=278145]: Done uploading segments... Nov 22 11:54:00.212200729 (315h49m58.763671191s) [upload=278145]: Upload finished. err: <nil> Duration: 5.828823651s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............X..............................................] duration: 3.52491347s trace: https://jaeger.dev.storj.io/trace/469b06ffc0b434d4 FAIL (2025-11-22 12:26:03 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.525008911s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.83753046s trace: https://jaeger.dev.storj.io/trace/76f1ecbff69b5d4e FAIL (2025-11-22 12:25:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, Piece ID: I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ) context deadline exceeded; piecestore: (Node ID: 12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, Piece ID: YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA) context deadline exceeded; piecestore: (Node ID: 1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, Piece ID: G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ) context deadline exceeded; piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA) context deadline exceeded ======LOGS====== nts... Nov 22 11:53:53.345386809 (315h49m51.89685727s) [upload=278145]: Got next segment. Inline: false Nov 22 11:53:53.345428137 (315h49m51.896898598s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 11:53:53.345433948 (315h49m51.896904419s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 11:53:54.603541081 (315h49m53.155011552s) [seg_pos={0 0}, upload=278145]: Begin upload segment... Nov 22 11:53:54.603595445 (315h49m53.155065916s) [seg_pos={0 0}, upload=278145]: Done begin upload segment. Nov 22 11:53:54.603668043 (315h49m53.155138494s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.603684554 (315h49m53.155155005s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604505828 (315h49m53.155976289s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604973038 (315h49m53.156443499s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.605425900 (315h49m53.156896361s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606323339 (315h49m53.15779381s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606878326 (315h49m53.158348777s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.607588267 (315h49m53.159058728s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610211763 (315h49m53.161682215s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610336751 (315h49m53.161807212s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612196994 (315h49m53.163667455s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612485593 (315h49m53.163956064s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614299567 (315h49m53.165770028s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614454202 (315h49m53.165924673s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.616397682 (315h49m53.167868143s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.621720077 (315h49m53.173190538s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:56.445932556 (315h49m54.997403007s) [upload=278145]: Next returned nil segment Nov 22 11:53:56.445949048 (315h49m54.997419509s) [upload=278145]: Waiting for error group managing segments... Nov 22 11:53:56.734588580 (315h49m55.286059041s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.796169791 (315h49m55.347640252s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.800552848 (315h49m55.352023309s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.802134119 (315h49m55.35360458s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.811866140 (315h49m55.363336591s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.820962632 (315h49m55.372433093s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.853755886 (315h49m55.405226347s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.857215724 (315h49m55.408686185s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.067648917 (315h49m55.619119378s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.093474313 (315h49m55.644944774s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.104509907 (315h49m55.655980378s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.505088550 (315h49m56.056559011s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609615419 (315h49m56.16108589s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609676946 (315h49m56.161147437s) [seg_pos={0 0}, upload=278145]: Segment reached optimal threshold of 13 pieces. Nov 22 11:53:57.609972077 (315h49m56.161442539s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection Nov 22 11:53:57.609982889 (315h49m56.16145336s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610025941 (315h49m56.161496432s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 22 11:53:57.610037101 (315h49m56.161507553s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610213978 (315h49m56.161684439s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection Nov 22 11:53:57.610224649 (315h49m56.16169511s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610261058 (315h49m56.16173152s) [seg_pos={0 0}, upload=278145]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 11:53:57.610421113 (315h49m56.161891584s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 11:53:57.610430852 (315h49m56.161901313s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 11:54:00.212174190 (315h49m58.763644651s) [upload=278145]: Done uploading segments... Nov 22 11:54:00.212200729 (315h49m58.763671191s) [upload=278145]: Upload finished. err: <nil> Duration: 5.837591445s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............X.............................................] duration: 3.457932825s trace: https://jaeger.dev.storj.io/trace/469b06ffc0b434d4 FAIL (2025-11-22 12:25:34 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.457961248s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............X............................................] duration: 3.638230836s trace: https://jaeger.dev.storj.io/trace/469b06ffc0b434d4 FAIL (2025-11-22 12:25:10 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.638280289s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 5.633935858s trace: https://jaeger.dev.storj.io/trace/76f1ecbff69b5d4e FAIL (2025-11-22 12:24:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, Piece ID: I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ) context deadline exceeded; piecestore: (Node ID: 12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, Piece ID: YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA) context deadline exceeded; piecestore: (Node ID: 1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, Piece ID: G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ) context deadline exceeded; piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA) context deadline exceeded ======LOGS====== nts... Nov 22 11:53:53.345386809 (315h49m51.89685727s) [upload=278145]: Got next segment. Inline: false Nov 22 11:53:53.345428137 (315h49m51.896898598s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 11:53:53.345433948 (315h49m51.896904419s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 11:53:54.603541081 (315h49m53.155011552s) [seg_pos={0 0}, upload=278145]: Begin upload segment... Nov 22 11:53:54.603595445 (315h49m53.155065916s) [seg_pos={0 0}, upload=278145]: Done begin upload segment. Nov 22 11:53:54.603668043 (315h49m53.155138494s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.603684554 (315h49m53.155155005s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604505828 (315h49m53.155976289s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604973038 (315h49m53.156443499s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.605425900 (315h49m53.156896361s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606323339 (315h49m53.15779381s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606878326 (315h49m53.158348777s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.607588267 (315h49m53.159058728s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610211763 (315h49m53.161682215s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610336751 (315h49m53.161807212s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612196994 (315h49m53.163667455s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612485593 (315h49m53.163956064s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614299567 (315h49m53.165770028s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614454202 (315h49m53.165924673s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.616397682 (315h49m53.167868143s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.621720077 (315h49m53.173190538s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:56.445932556 (315h49m54.997403007s) [upload=278145]: Next returned nil segment Nov 22 11:53:56.445949048 (315h49m54.997419509s) [upload=278145]: Waiting for error group managing segments... Nov 22 11:53:56.734588580 (315h49m55.286059041s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.796169791 (315h49m55.347640252s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.800552848 (315h49m55.352023309s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.802134119 (315h49m55.35360458s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.811866140 (315h49m55.363336591s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.820962632 (315h49m55.372433093s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.853755886 (315h49m55.405226347s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.857215724 (315h49m55.408686185s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.067648917 (315h49m55.619119378s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.093474313 (315h49m55.644944774s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.104509907 (315h49m55.655980378s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.505088550 (315h49m56.056559011s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609615419 (315h49m56.16108589s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609676946 (315h49m56.161147437s) [seg_pos={0 0}, upload=278145]: Segment reached optimal threshold of 13 pieces. Nov 22 11:53:57.609972077 (315h49m56.161442539s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection Nov 22 11:53:57.609982889 (315h49m56.16145336s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610025941 (315h49m56.161496432s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 22 11:53:57.610037101 (315h49m56.161507553s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610213978 (315h49m56.161684439s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection Nov 22 11:53:57.610224649 (315h49m56.16169511s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610261058 (315h49m56.16173152s) [seg_pos={0 0}, upload=278145]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 11:53:57.610421113 (315h49m56.161891584s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 11:53:57.610430852 (315h49m56.161901313s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 11:54:00.212174190 (315h49m58.763644651s) [upload=278145]: Done uploading segments... Nov 22 11:54:00.212200729 (315h49m58.763671191s) [upload=278145]: Upload finished. err: <nil> Duration: 5.634038014s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.685049672s trace: https://jaeger.dev.storj.io/trace/76f1ecbff69b5d4e FAIL (2025-11-22 12:24:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, Piece ID: I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ) context deadline exceeded; piecestore: (Node ID: 12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, Piece ID: YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA) context deadline exceeded; piecestore: (Node ID: 1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, Piece ID: G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ) context deadline exceeded; piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA) context deadline exceeded ======LOGS====== nts... Nov 22 11:53:53.345386809 (315h49m51.89685727s) [upload=278145]: Got next segment. Inline: false Nov 22 11:53:53.345428137 (315h49m51.896898598s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 11:53:53.345433948 (315h49m51.896904419s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 11:53:54.603541081 (315h49m53.155011552s) [seg_pos={0 0}, upload=278145]: Begin upload segment... Nov 22 11:53:54.603595445 (315h49m53.155065916s) [seg_pos={0 0}, upload=278145]: Done begin upload segment. Nov 22 11:53:54.603668043 (315h49m53.155138494s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.603684554 (315h49m53.155155005s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604505828 (315h49m53.155976289s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604973038 (315h49m53.156443499s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.605425900 (315h49m53.156896361s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606323339 (315h49m53.15779381s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606878326 (315h49m53.158348777s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.607588267 (315h49m53.159058728s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610211763 (315h49m53.161682215s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610336751 (315h49m53.161807212s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612196994 (315h49m53.163667455s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612485593 (315h49m53.163956064s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614299567 (315h49m53.165770028s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614454202 (315h49m53.165924673s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.616397682 (315h49m53.167868143s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.621720077 (315h49m53.173190538s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:56.445932556 (315h49m54.997403007s) [upload=278145]: Next returned nil segment Nov 22 11:53:56.445949048 (315h49m54.997419509s) [upload=278145]: Waiting for error group managing segments... Nov 22 11:53:56.734588580 (315h49m55.286059041s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.796169791 (315h49m55.347640252s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.800552848 (315h49m55.352023309s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.802134119 (315h49m55.35360458s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.811866140 (315h49m55.363336591s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.820962632 (315h49m55.372433093s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.853755886 (315h49m55.405226347s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.857215724 (315h49m55.408686185s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.067648917 (315h49m55.619119378s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.093474313 (315h49m55.644944774s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.104509907 (315h49m55.655980378s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.505088550 (315h49m56.056559011s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609615419 (315h49m56.16108589s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609676946 (315h49m56.161147437s) [seg_pos={0 0}, upload=278145]: Segment reached optimal threshold of 13 pieces. Nov 22 11:53:57.609972077 (315h49m56.161442539s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection Nov 22 11:53:57.609982889 (315h49m56.16145336s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610025941 (315h49m56.161496432s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 22 11:53:57.610037101 (315h49m56.161507553s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610213978 (315h49m56.161684439s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection Nov 22 11:53:57.610224649 (315h49m56.16169511s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610261058 (315h49m56.16173152s) [seg_pos={0 0}, upload=278145]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 11:53:57.610421113 (315h49m56.161891584s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 11:53:57.610430852 (315h49m56.161901313s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 11:54:00.212174190 (315h49m58.763644651s) [upload=278145]: Done uploading segments... Nov 22 11:54:00.212200729 (315h49m58.763671191s) [upload=278145]: Upload finished. err: <nil> Duration: 5.685078837s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................X...........................................] duration: 3.400965468s trace: https://jaeger.dev.storj.io/trace/469b06ffc0b434d4 FAIL (2025-11-22 12:24:32 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.40099794s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................X..........................................] duration: 3.459195679s trace: https://jaeger.dev.storj.io/trace/469b06ffc0b434d4 FAIL (2025-11-22 12:23:34 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.45922192s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.684049643s trace: https://jaeger.dev.storj.io/trace/76f1ecbff69b5d4e FAIL (2025-11-22 12:23:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, Piece ID: I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ) context deadline exceeded; piecestore: (Node ID: 12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, Piece ID: YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA) context deadline exceeded; piecestore: (Node ID: 1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, Piece ID: G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ) context deadline exceeded; piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA) context deadline exceeded ======LOGS====== nts... Nov 22 11:53:53.345386809 (315h49m51.89685727s) [upload=278145]: Got next segment. Inline: false Nov 22 11:53:53.345428137 (315h49m51.896898598s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 11:53:53.345433948 (315h49m51.896904419s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 11:53:54.603541081 (315h49m53.155011552s) [seg_pos={0 0}, upload=278145]: Begin upload segment... Nov 22 11:53:54.603595445 (315h49m53.155065916s) [seg_pos={0 0}, upload=278145]: Done begin upload segment. Nov 22 11:53:54.603668043 (315h49m53.155138494s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.603684554 (315h49m53.155155005s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604505828 (315h49m53.155976289s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604973038 (315h49m53.156443499s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.605425900 (315h49m53.156896361s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606323339 (315h49m53.15779381s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606878326 (315h49m53.158348777s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.607588267 (315h49m53.159058728s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610211763 (315h49m53.161682215s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610336751 (315h49m53.161807212s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612196994 (315h49m53.163667455s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612485593 (315h49m53.163956064s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614299567 (315h49m53.165770028s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614454202 (315h49m53.165924673s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.616397682 (315h49m53.167868143s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.621720077 (315h49m53.173190538s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:56.445932556 (315h49m54.997403007s) [upload=278145]: Next returned nil segment Nov 22 11:53:56.445949048 (315h49m54.997419509s) [upload=278145]: Waiting for error group managing segments... Nov 22 11:53:56.734588580 (315h49m55.286059041s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.796169791 (315h49m55.347640252s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.800552848 (315h49m55.352023309s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.802134119 (315h49m55.35360458s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.811866140 (315h49m55.363336591s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.820962632 (315h49m55.372433093s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.853755886 (315h49m55.405226347s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.857215724 (315h49m55.408686185s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.067648917 (315h49m55.619119378s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.093474313 (315h49m55.644944774s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.104509907 (315h49m55.655980378s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.505088550 (315h49m56.056559011s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609615419 (315h49m56.16108589s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609676946 (315h49m56.161147437s) [seg_pos={0 0}, upload=278145]: Segment reached optimal threshold of 13 pieces. Nov 22 11:53:57.609972077 (315h49m56.161442539s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection Nov 22 11:53:57.609982889 (315h49m56.16145336s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610025941 (315h49m56.161496432s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 22 11:53:57.610037101 (315h49m56.161507553s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610213978 (315h49m56.161684439s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection Nov 22 11:53:57.610224649 (315h49m56.16169511s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610261058 (315h49m56.16173152s) [seg_pos={0 0}, upload=278145]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 11:53:57.610421113 (315h49m56.161891584s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 11:53:57.610430852 (315h49m56.161901313s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 11:54:00.212174190 (315h49m58.763644651s) [upload=278145]: Done uploading segments... Nov 22 11:54:00.212200729 (315h49m58.763671191s) [upload=278145]: Upload finished. err: <nil> Duration: 5.684245867s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.688813574s trace: https://jaeger.dev.storj.io/trace/76f1ecbff69b5d4e FAIL (2025-11-22 12:23:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, Piece ID: I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ) context deadline exceeded; piecestore: (Node ID: 12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, Piece ID: YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA) context deadline exceeded; piecestore: (Node ID: 1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, Piece ID: G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ) context deadline exceeded; piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA) context deadline exceeded ======LOGS====== nts... Nov 22 11:53:53.345386809 (315h49m51.89685727s) [upload=278145]: Got next segment. Inline: false Nov 22 11:53:53.345428137 (315h49m51.896898598s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 11:53:53.345433948 (315h49m51.896904419s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 11:53:54.603541081 (315h49m53.155011552s) [seg_pos={0 0}, upload=278145]: Begin upload segment... Nov 22 11:53:54.603595445 (315h49m53.155065916s) [seg_pos={0 0}, upload=278145]: Done begin upload segment. Nov 22 11:53:54.603668043 (315h49m53.155138494s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.603684554 (315h49m53.155155005s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604505828 (315h49m53.155976289s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604973038 (315h49m53.156443499s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.605425900 (315h49m53.156896361s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606323339 (315h49m53.15779381s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606878326 (315h49m53.158348777s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.607588267 (315h49m53.159058728s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610211763 (315h49m53.161682215s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610336751 (315h49m53.161807212s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612196994 (315h49m53.163667455s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612485593 (315h49m53.163956064s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614299567 (315h49m53.165770028s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614454202 (315h49m53.165924673s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.616397682 (315h49m53.167868143s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.621720077 (315h49m53.173190538s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:56.445932556 (315h49m54.997403007s) [upload=278145]: Next returned nil segment Nov 22 11:53:56.445949048 (315h49m54.997419509s) [upload=278145]: Waiting for error group managing segments... Nov 22 11:53:56.734588580 (315h49m55.286059041s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.796169791 (315h49m55.347640252s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.800552848 (315h49m55.352023309s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.802134119 (315h49m55.35360458s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.811866140 (315h49m55.363336591s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.820962632 (315h49m55.372433093s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.853755886 (315h49m55.405226347s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.857215724 (315h49m55.408686185s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.067648917 (315h49m55.619119378s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.093474313 (315h49m55.644944774s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.104509907 (315h49m55.655980378s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.505088550 (315h49m56.056559011s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609615419 (315h49m56.16108589s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609676946 (315h49m56.161147437s) [seg_pos={0 0}, upload=278145]: Segment reached optimal threshold of 13 pieces. Nov 22 11:53:57.609972077 (315h49m56.161442539s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection Nov 22 11:53:57.609982889 (315h49m56.16145336s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610025941 (315h49m56.161496432s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 22 11:53:57.610037101 (315h49m56.161507553s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610213978 (315h49m56.161684439s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection Nov 22 11:53:57.610224649 (315h49m56.16169511s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610261058 (315h49m56.16173152s) [seg_pos={0 0}, upload=278145]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 11:53:57.610421113 (315h49m56.161891584s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 11:53:57.610430852 (315h49m56.161901313s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 11:54:00.212174190 (315h49m58.763644651s) [upload=278145]: Done uploading segments... Nov 22 11:54:00.212200729 (315h49m58.763671191s) [upload=278145]: Upload finished. err: <nil> Duration: 5.688839414s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.859087259s trace: https://jaeger.dev.storj.io/trace/76f1ecbff69b5d4e FAIL (2025-11-22 12:22:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, Piece ID: I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ) context deadline exceeded; piecestore: (Node ID: 12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, Piece ID: YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA) context deadline exceeded; piecestore: (Node ID: 1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, Piece ID: G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ) context deadline exceeded; piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA) context deadline exceeded ======LOGS====== nts... Nov 22 11:53:53.345386809 (315h49m51.89685727s) [upload=278145]: Got next segment. Inline: false Nov 22 11:53:53.345428137 (315h49m51.896898598s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 11:53:53.345433948 (315h49m51.896904419s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 11:53:54.603541081 (315h49m53.155011552s) [seg_pos={0 0}, upload=278145]: Begin upload segment... Nov 22 11:53:54.603595445 (315h49m53.155065916s) [seg_pos={0 0}, upload=278145]: Done begin upload segment. Nov 22 11:53:54.603668043 (315h49m53.155138494s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.603684554 (315h49m53.155155005s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604505828 (315h49m53.155976289s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604973038 (315h49m53.156443499s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.605425900 (315h49m53.156896361s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606323339 (315h49m53.15779381s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606878326 (315h49m53.158348777s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.607588267 (315h49m53.159058728s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610211763 (315h49m53.161682215s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610336751 (315h49m53.161807212s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612196994 (315h49m53.163667455s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612485593 (315h49m53.163956064s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614299567 (315h49m53.165770028s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614454202 (315h49m53.165924673s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.616397682 (315h49m53.167868143s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.621720077 (315h49m53.173190538s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:56.445932556 (315h49m54.997403007s) [upload=278145]: Next returned nil segment Nov 22 11:53:56.445949048 (315h49m54.997419509s) [upload=278145]: Waiting for error group managing segments... Nov 22 11:53:56.734588580 (315h49m55.286059041s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.796169791 (315h49m55.347640252s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.800552848 (315h49m55.352023309s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.802134119 (315h49m55.35360458s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.811866140 (315h49m55.363336591s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.820962632 (315h49m55.372433093s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.853755886 (315h49m55.405226347s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.857215724 (315h49m55.408686185s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.067648917 (315h49m55.619119378s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.093474313 (315h49m55.644944774s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.104509907 (315h49m55.655980378s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.505088550 (315h49m56.056559011s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609615419 (315h49m56.16108589s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609676946 (315h49m56.161147437s) [seg_pos={0 0}, upload=278145]: Segment reached optimal threshold of 13 pieces. Nov 22 11:53:57.609972077 (315h49m56.161442539s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection Nov 22 11:53:57.609982889 (315h49m56.16145336s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610025941 (315h49m56.161496432s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 22 11:53:57.610037101 (315h49m56.161507553s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610213978 (315h49m56.161684439s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection Nov 22 11:53:57.610224649 (315h49m56.16169511s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610261058 (315h49m56.16173152s) [seg_pos={0 0}, upload=278145]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 11:53:57.610421113 (315h49m56.161891584s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 11:53:57.610430852 (315h49m56.161901313s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 11:54:00.212174190 (315h49m58.763644651s) [upload=278145]: Done uploading segments... Nov 22 11:54:00.212200729 (315h49m58.763671191s) [upload=278145]: Upload finished. err: <nil> Duration: 5.859116464s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................X.........................................] duration: 3.420241785s trace: https://jaeger.dev.storj.io/trace/469b06ffc0b434d4 FAIL (2025-11-22 12:22:34 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.420262616s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.882988459s trace: https://jaeger.dev.storj.io/trace/76f1ecbff69b5d4e FAIL (2025-11-22 12:22:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, Piece ID: I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ) context deadline exceeded; piecestore: (Node ID: 12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, Piece ID: YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA) context deadline exceeded; piecestore: (Node ID: 1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, Piece ID: G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ) context deadline exceeded; piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA) context deadline exceeded ======LOGS====== nts... Nov 22 11:53:53.345386809 (315h49m51.89685727s) [upload=278145]: Got next segment. Inline: false Nov 22 11:53:53.345428137 (315h49m51.896898598s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 11:53:53.345433948 (315h49m51.896904419s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 11:53:54.603541081 (315h49m53.155011552s) [seg_pos={0 0}, upload=278145]: Begin upload segment... Nov 22 11:53:54.603595445 (315h49m53.155065916s) [seg_pos={0 0}, upload=278145]: Done begin upload segment. Nov 22 11:53:54.603668043 (315h49m53.155138494s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.603684554 (315h49m53.155155005s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604505828 (315h49m53.155976289s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604973038 (315h49m53.156443499s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.605425900 (315h49m53.156896361s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606323339 (315h49m53.15779381s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606878326 (315h49m53.158348777s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.607588267 (315h49m53.159058728s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610211763 (315h49m53.161682215s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610336751 (315h49m53.161807212s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612196994 (315h49m53.163667455s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612485593 (315h49m53.163956064s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614299567 (315h49m53.165770028s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614454202 (315h49m53.165924673s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.616397682 (315h49m53.167868143s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.621720077 (315h49m53.173190538s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:56.445932556 (315h49m54.997403007s) [upload=278145]: Next returned nil segment Nov 22 11:53:56.445949048 (315h49m54.997419509s) [upload=278145]: Waiting for error group managing segments... Nov 22 11:53:56.734588580 (315h49m55.286059041s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.796169791 (315h49m55.347640252s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.800552848 (315h49m55.352023309s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.802134119 (315h49m55.35360458s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.811866140 (315h49m55.363336591s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.820962632 (315h49m55.372433093s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.853755886 (315h49m55.405226347s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.857215724 (315h49m55.408686185s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.067648917 (315h49m55.619119378s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.093474313 (315h49m55.644944774s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.104509907 (315h49m55.655980378s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.505088550 (315h49m56.056559011s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609615419 (315h49m56.16108589s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609676946 (315h49m56.161147437s) [seg_pos={0 0}, upload=278145]: Segment reached optimal threshold of 13 pieces. Nov 22 11:53:57.609972077 (315h49m56.161442539s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection Nov 22 11:53:57.609982889 (315h49m56.16145336s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610025941 (315h49m56.161496432s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 22 11:53:57.610037101 (315h49m56.161507553s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610213978 (315h49m56.161684439s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection Nov 22 11:53:57.610224649 (315h49m56.16169511s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610261058 (315h49m56.16173152s) [seg_pos={0 0}, upload=278145]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 11:53:57.610421113 (315h49m56.161891584s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 11:53:57.610430852 (315h49m56.161901313s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 11:54:00.212174190 (315h49m58.763644651s) [upload=278145]: Done uploading segments... Nov 22 11:54:00.212200729 (315h49m58.763671191s) [upload=278145]: Upload finished. err: <nil> Duration: 5.883009088s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.750919816s trace: https://jaeger.dev.storj.io/trace/76f1ecbff69b5d4e FAIL (2025-11-22 12:21:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, Piece ID: I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ) context deadline exceeded; piecestore: (Node ID: 12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, Piece ID: YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA) context deadline exceeded; piecestore: (Node ID: 1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, Piece ID: G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ) context deadline exceeded; piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA) context deadline exceeded ======LOGS====== nts... Nov 22 11:53:53.345386809 (315h49m51.89685727s) [upload=278145]: Got next segment. Inline: false Nov 22 11:53:53.345428137 (315h49m51.896898598s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 11:53:53.345433948 (315h49m51.896904419s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 11:53:54.603541081 (315h49m53.155011552s) [seg_pos={0 0}, upload=278145]: Begin upload segment... Nov 22 11:53:54.603595445 (315h49m53.155065916s) [seg_pos={0 0}, upload=278145]: Done begin upload segment. Nov 22 11:53:54.603668043 (315h49m53.155138494s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.603684554 (315h49m53.155155005s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604505828 (315h49m53.155976289s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604973038 (315h49m53.156443499s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.605425900 (315h49m53.156896361s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606323339 (315h49m53.15779381s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606878326 (315h49m53.158348777s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.607588267 (315h49m53.159058728s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610211763 (315h49m53.161682215s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610336751 (315h49m53.161807212s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612196994 (315h49m53.163667455s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612485593 (315h49m53.163956064s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614299567 (315h49m53.165770028s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614454202 (315h49m53.165924673s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.616397682 (315h49m53.167868143s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.621720077 (315h49m53.173190538s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:56.445932556 (315h49m54.997403007s) [upload=278145]: Next returned nil segment Nov 22 11:53:56.445949048 (315h49m54.997419509s) [upload=278145]: Waiting for error group managing segments... Nov 22 11:53:56.734588580 (315h49m55.286059041s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.796169791 (315h49m55.347640252s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.800552848 (315h49m55.352023309s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.802134119 (315h49m55.35360458s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.811866140 (315h49m55.363336591s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.820962632 (315h49m55.372433093s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.853755886 (315h49m55.405226347s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.857215724 (315h49m55.408686185s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.067648917 (315h49m55.619119378s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.093474313 (315h49m55.644944774s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.104509907 (315h49m55.655980378s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.505088550 (315h49m56.056559011s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609615419 (315h49m56.16108589s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609676946 (315h49m56.161147437s) [seg_pos={0 0}, upload=278145]: Segment reached optimal threshold of 13 pieces. Nov 22 11:53:57.609972077 (315h49m56.161442539s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection Nov 22 11:53:57.609982889 (315h49m56.16145336s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610025941 (315h49m56.161496432s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 22 11:53:57.610037101 (315h49m56.161507553s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610213978 (315h49m56.161684439s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection Nov 22 11:53:57.610224649 (315h49m56.16169511s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610261058 (315h49m56.16173152s) [seg_pos={0 0}, upload=278145]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 11:53:57.610421113 (315h49m56.161891584s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 11:53:57.610430852 (315h49m56.161901313s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 11:54:00.212174190 (315h49m58.763644651s) [upload=278145]: Done uploading segments... Nov 22 11:54:00.212200729 (315h49m58.763671191s) [upload=278145]: Upload finished. err: <nil> Duration: 5.75095812s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................X........................................] duration: 3.409736451s trace: https://jaeger.dev.storj.io/trace/469b06ffc0b434d4 FAIL (2025-11-22 12:21:42 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.409797106s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 5.756626373s trace: https://jaeger.dev.storj.io/trace/76f1ecbff69b5d4e FAIL (2025-11-22 12:21:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, Piece ID: I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ) context deadline exceeded; piecestore: (Node ID: 12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, Piece ID: YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA) context deadline exceeded; piecestore: (Node ID: 1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, Piece ID: G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ) context deadline exceeded; piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA) context deadline exceeded ======LOGS====== nts... Nov 22 11:53:53.345386809 (315h49m51.89685727s) [upload=278145]: Got next segment. Inline: false Nov 22 11:53:53.345428137 (315h49m51.896898598s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 11:53:53.345433948 (315h49m51.896904419s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 11:53:54.603541081 (315h49m53.155011552s) [seg_pos={0 0}, upload=278145]: Begin upload segment... Nov 22 11:53:54.603595445 (315h49m53.155065916s) [seg_pos={0 0}, upload=278145]: Done begin upload segment. Nov 22 11:53:54.603668043 (315h49m53.155138494s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.603684554 (315h49m53.155155005s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604505828 (315h49m53.155976289s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604973038 (315h49m53.156443499s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.605425900 (315h49m53.156896361s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606323339 (315h49m53.15779381s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606878326 (315h49m53.158348777s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.607588267 (315h49m53.159058728s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610211763 (315h49m53.161682215s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610336751 (315h49m53.161807212s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612196994 (315h49m53.163667455s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612485593 (315h49m53.163956064s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614299567 (315h49m53.165770028s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614454202 (315h49m53.165924673s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.616397682 (315h49m53.167868143s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.621720077 (315h49m53.173190538s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:56.445932556 (315h49m54.997403007s) [upload=278145]: Next returned nil segment Nov 22 11:53:56.445949048 (315h49m54.997419509s) [upload=278145]: Waiting for error group managing segments... Nov 22 11:53:56.734588580 (315h49m55.286059041s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.796169791 (315h49m55.347640252s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.800552848 (315h49m55.352023309s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.802134119 (315h49m55.35360458s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.811866140 (315h49m55.363336591s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.820962632 (315h49m55.372433093s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.853755886 (315h49m55.405226347s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.857215724 (315h49m55.408686185s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.067648917 (315h49m55.619119378s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.093474313 (315h49m55.644944774s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.104509907 (315h49m55.655980378s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.505088550 (315h49m56.056559011s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609615419 (315h49m56.16108589s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609676946 (315h49m56.161147437s) [seg_pos={0 0}, upload=278145]: Segment reached optimal threshold of 13 pieces. Nov 22 11:53:57.609972077 (315h49m56.161442539s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection Nov 22 11:53:57.609982889 (315h49m56.16145336s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610025941 (315h49m56.161496432s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 22 11:53:57.610037101 (315h49m56.161507553s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610213978 (315h49m56.161684439s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection Nov 22 11:53:57.610224649 (315h49m56.16169511s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610261058 (315h49m56.16173152s) [seg_pos={0 0}, upload=278145]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 11:53:57.610421113 (315h49m56.161891584s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 11:53:57.610430852 (315h49m56.161901313s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 11:54:00.212174190 (315h49m58.763644651s) [upload=278145]: Done uploading segments... Nov 22 11:54:00.212200729 (315h49m58.763671191s) [upload=278145]: Upload finished. err: <nil> Duration: 5.756646171s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 5.646529787s trace: https://jaeger.dev.storj.io/trace/76f1ecbff69b5d4e FAIL (2025-11-22 12:20:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, Piece ID: I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ) context deadline exceeded; piecestore: (Node ID: 12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, Piece ID: YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA) context deadline exceeded; piecestore: (Node ID: 1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, Piece ID: G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ) context deadline exceeded; piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA) context deadline exceeded ======LOGS====== nts... Nov 22 11:53:53.345386809 (315h49m51.89685727s) [upload=278145]: Got next segment. Inline: false Nov 22 11:53:53.345428137 (315h49m51.896898598s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 11:53:53.345433948 (315h49m51.896904419s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 11:53:54.603541081 (315h49m53.155011552s) [seg_pos={0 0}, upload=278145]: Begin upload segment... Nov 22 11:53:54.603595445 (315h49m53.155065916s) [seg_pos={0 0}, upload=278145]: Done begin upload segment. Nov 22 11:53:54.603668043 (315h49m53.155138494s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.603684554 (315h49m53.155155005s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604505828 (315h49m53.155976289s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604973038 (315h49m53.156443499s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.605425900 (315h49m53.156896361s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606323339 (315h49m53.15779381s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606878326 (315h49m53.158348777s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.607588267 (315h49m53.159058728s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610211763 (315h49m53.161682215s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610336751 (315h49m53.161807212s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612196994 (315h49m53.163667455s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612485593 (315h49m53.163956064s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614299567 (315h49m53.165770028s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614454202 (315h49m53.165924673s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.616397682 (315h49m53.167868143s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.621720077 (315h49m53.173190538s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:56.445932556 (315h49m54.997403007s) [upload=278145]: Next returned nil segment Nov 22 11:53:56.445949048 (315h49m54.997419509s) [upload=278145]: Waiting for error group managing segments... Nov 22 11:53:56.734588580 (315h49m55.286059041s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.796169791 (315h49m55.347640252s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.800552848 (315h49m55.352023309s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.802134119 (315h49m55.35360458s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.811866140 (315h49m55.363336591s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.820962632 (315h49m55.372433093s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.853755886 (315h49m55.405226347s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.857215724 (315h49m55.408686185s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.067648917 (315h49m55.619119378s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.093474313 (315h49m55.644944774s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.104509907 (315h49m55.655980378s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.505088550 (315h49m56.056559011s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609615419 (315h49m56.16108589s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609676946 (315h49m56.161147437s) [seg_pos={0 0}, upload=278145]: Segment reached optimal threshold of 13 pieces. Nov 22 11:53:57.609972077 (315h49m56.161442539s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection Nov 22 11:53:57.609982889 (315h49m56.16145336s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610025941 (315h49m56.161496432s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 22 11:53:57.610037101 (315h49m56.161507553s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610213978 (315h49m56.161684439s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection Nov 22 11:53:57.610224649 (315h49m56.16169511s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610261058 (315h49m56.16173152s) [seg_pos={0 0}, upload=278145]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 11:53:57.610421113 (315h49m56.161891584s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 11:53:57.610430852 (315h49m56.161901313s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 11:54:00.212174190 (315h49m58.763644651s) [upload=278145]: Done uploading segments... Nov 22 11:54:00.212200729 (315h49m58.763671191s) [upload=278145]: Upload finished. err: <nil> Duration: 5.646553894s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................X.......................................] duration: 3.527375186s trace: https://jaeger.dev.storj.io/trace/469b06ffc0b434d4 FAIL (2025-11-22 12:20:38 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.527397718s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.770275081s trace: https://jaeger.dev.storj.io/trace/76f1ecbff69b5d4e FAIL (2025-11-22 12:19:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, Piece ID: I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ) context deadline exceeded; piecestore: (Node ID: 12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, Piece ID: YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA) context deadline exceeded; piecestore: (Node ID: 1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, Piece ID: G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ) context deadline exceeded; piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA) context deadline exceeded ======LOGS====== nts... Nov 22 11:53:53.345386809 (315h49m51.89685727s) [upload=278145]: Got next segment. Inline: false Nov 22 11:53:53.345428137 (315h49m51.896898598s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 11:53:53.345433948 (315h49m51.896904419s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 11:53:54.603541081 (315h49m53.155011552s) [seg_pos={0 0}, upload=278145]: Begin upload segment... Nov 22 11:53:54.603595445 (315h49m53.155065916s) [seg_pos={0 0}, upload=278145]: Done begin upload segment. Nov 22 11:53:54.603668043 (315h49m53.155138494s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.603684554 (315h49m53.155155005s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604505828 (315h49m53.155976289s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604973038 (315h49m53.156443499s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.605425900 (315h49m53.156896361s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606323339 (315h49m53.15779381s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606878326 (315h49m53.158348777s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.607588267 (315h49m53.159058728s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610211763 (315h49m53.161682215s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610336751 (315h49m53.161807212s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612196994 (315h49m53.163667455s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612485593 (315h49m53.163956064s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614299567 (315h49m53.165770028s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614454202 (315h49m53.165924673s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.616397682 (315h49m53.167868143s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.621720077 (315h49m53.173190538s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:56.445932556 (315h49m54.997403007s) [upload=278145]: Next returned nil segment Nov 22 11:53:56.445949048 (315h49m54.997419509s) [upload=278145]: Waiting for error group managing segments... Nov 22 11:53:56.734588580 (315h49m55.286059041s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.796169791 (315h49m55.347640252s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.800552848 (315h49m55.352023309s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.802134119 (315h49m55.35360458s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.811866140 (315h49m55.363336591s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.820962632 (315h49m55.372433093s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.853755886 (315h49m55.405226347s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.857215724 (315h49m55.408686185s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.067648917 (315h49m55.619119378s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.093474313 (315h49m55.644944774s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.104509907 (315h49m55.655980378s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.505088550 (315h49m56.056559011s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609615419 (315h49m56.16108589s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609676946 (315h49m56.161147437s) [seg_pos={0 0}, upload=278145]: Segment reached optimal threshold of 13 pieces. Nov 22 11:53:57.609972077 (315h49m56.161442539s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection Nov 22 11:53:57.609982889 (315h49m56.16145336s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610025941 (315h49m56.161496432s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 22 11:53:57.610037101 (315h49m56.161507553s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610213978 (315h49m56.161684439s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection Nov 22 11:53:57.610224649 (315h49m56.16169511s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610261058 (315h49m56.16173152s) [seg_pos={0 0}, upload=278145]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 11:53:57.610421113 (315h49m56.161891584s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 11:53:57.610430852 (315h49m56.161901313s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 11:54:00.212174190 (315h49m58.763644651s) [upload=278145]: Done uploading segments... Nov 22 11:54:00.212200729 (315h49m58.763671191s) [upload=278145]: Upload finished. err: <nil> Duration: 5.770307543s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................X......................................] duration: 3.453994336s trace: https://jaeger.dev.storj.io/trace/469b06ffc0b434d4 FAIL (2025-11-22 12:19:33 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.454019544s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.668790504s trace: https://jaeger.dev.storj.io/trace/76f1ecbff69b5d4e FAIL (2025-11-22 12:19:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, Piece ID: I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ) context deadline exceeded; piecestore: (Node ID: 12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, Piece ID: YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA) context deadline exceeded; piecestore: (Node ID: 1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, Piece ID: G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ) context deadline exceeded; piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA) context deadline exceeded ======LOGS====== nts... Nov 22 11:53:53.345386809 (315h49m51.89685727s) [upload=278145]: Got next segment. Inline: false Nov 22 11:53:53.345428137 (315h49m51.896898598s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 11:53:53.345433948 (315h49m51.896904419s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 11:53:54.603541081 (315h49m53.155011552s) [seg_pos={0 0}, upload=278145]: Begin upload segment... Nov 22 11:53:54.603595445 (315h49m53.155065916s) [seg_pos={0 0}, upload=278145]: Done begin upload segment. Nov 22 11:53:54.603668043 (315h49m53.155138494s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.603684554 (315h49m53.155155005s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604505828 (315h49m53.155976289s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604973038 (315h49m53.156443499s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.605425900 (315h49m53.156896361s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606323339 (315h49m53.15779381s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606878326 (315h49m53.158348777s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.607588267 (315h49m53.159058728s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610211763 (315h49m53.161682215s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610336751 (315h49m53.161807212s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612196994 (315h49m53.163667455s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612485593 (315h49m53.163956064s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614299567 (315h49m53.165770028s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614454202 (315h49m53.165924673s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.616397682 (315h49m53.167868143s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.621720077 (315h49m53.173190538s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:56.445932556 (315h49m54.997403007s) [upload=278145]: Next returned nil segment Nov 22 11:53:56.445949048 (315h49m54.997419509s) [upload=278145]: Waiting for error group managing segments... Nov 22 11:53:56.734588580 (315h49m55.286059041s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.796169791 (315h49m55.347640252s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.800552848 (315h49m55.352023309s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.802134119 (315h49m55.35360458s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.811866140 (315h49m55.363336591s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.820962632 (315h49m55.372433093s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.853755886 (315h49m55.405226347s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.857215724 (315h49m55.408686185s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.067648917 (315h49m55.619119378s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.093474313 (315h49m55.644944774s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.104509907 (315h49m55.655980378s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.505088550 (315h49m56.056559011s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609615419 (315h49m56.16108589s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609676946 (315h49m56.161147437s) [seg_pos={0 0}, upload=278145]: Segment reached optimal threshold of 13 pieces. Nov 22 11:53:57.609972077 (315h49m56.161442539s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection Nov 22 11:53:57.609982889 (315h49m56.16145336s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610025941 (315h49m56.161496432s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 22 11:53:57.610037101 (315h49m56.161507553s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610213978 (315h49m56.161684439s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection Nov 22 11:53:57.610224649 (315h49m56.16169511s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610261058 (315h49m56.16173152s) [seg_pos={0 0}, upload=278145]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 11:53:57.610421113 (315h49m56.161891584s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 11:53:57.610430852 (315h49m56.161901313s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 11:54:00.212174190 (315h49m58.763644651s) [upload=278145]: Done uploading segments... Nov 22 11:54:00.212200729 (315h49m58.763671191s) [upload=278145]: Upload finished. err: <nil> Duration: 5.668822865s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................X.....................................] duration: 3.4602148s trace: https://jaeger.dev.storj.io/trace/469b06ffc0b434d4 FAIL (2025-11-22 12:18:46 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.460242193s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 5.776119571s trace: https://jaeger.dev.storj.io/trace/76f1ecbff69b5d4e FAIL (2025-11-22 12:18:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, Piece ID: I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ) context deadline exceeded; piecestore: (Node ID: 12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, Piece ID: YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA) context deadline exceeded; piecestore: (Node ID: 1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, Piece ID: G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ) context deadline exceeded; piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA) context deadline exceeded ======LOGS====== nts... Nov 22 11:53:53.345386809 (315h49m51.89685727s) [upload=278145]: Got next segment. Inline: false Nov 22 11:53:53.345428137 (315h49m51.896898598s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 11:53:53.345433948 (315h49m51.896904419s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 11:53:54.603541081 (315h49m53.155011552s) [seg_pos={0 0}, upload=278145]: Begin upload segment... Nov 22 11:53:54.603595445 (315h49m53.155065916s) [seg_pos={0 0}, upload=278145]: Done begin upload segment. Nov 22 11:53:54.603668043 (315h49m53.155138494s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.603684554 (315h49m53.155155005s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604505828 (315h49m53.155976289s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604973038 (315h49m53.156443499s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.605425900 (315h49m53.156896361s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606323339 (315h49m53.15779381s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606878326 (315h49m53.158348777s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.607588267 (315h49m53.159058728s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610211763 (315h49m53.161682215s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610336751 (315h49m53.161807212s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612196994 (315h49m53.163667455s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612485593 (315h49m53.163956064s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614299567 (315h49m53.165770028s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614454202 (315h49m53.165924673s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.616397682 (315h49m53.167868143s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.621720077 (315h49m53.173190538s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:56.445932556 (315h49m54.997403007s) [upload=278145]: Next returned nil segment Nov 22 11:53:56.445949048 (315h49m54.997419509s) [upload=278145]: Waiting for error group managing segments... Nov 22 11:53:56.734588580 (315h49m55.286059041s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.796169791 (315h49m55.347640252s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.800552848 (315h49m55.352023309s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.802134119 (315h49m55.35360458s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.811866140 (315h49m55.363336591s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.820962632 (315h49m55.372433093s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.853755886 (315h49m55.405226347s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.857215724 (315h49m55.408686185s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.067648917 (315h49m55.619119378s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.093474313 (315h49m55.644944774s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.104509907 (315h49m55.655980378s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.505088550 (315h49m56.056559011s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609615419 (315h49m56.16108589s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609676946 (315h49m56.161147437s) [seg_pos={0 0}, upload=278145]: Segment reached optimal threshold of 13 pieces. Nov 22 11:53:57.609972077 (315h49m56.161442539s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection Nov 22 11:53:57.609982889 (315h49m56.16145336s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610025941 (315h49m56.161496432s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 22 11:53:57.610037101 (315h49m56.161507553s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610213978 (315h49m56.161684439s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection Nov 22 11:53:57.610224649 (315h49m56.16169511s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610261058 (315h49m56.16173152s) [seg_pos={0 0}, upload=278145]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 11:53:57.610421113 (315h49m56.161891584s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 11:53:57.610430852 (315h49m56.161901313s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 11:54:00.212174190 (315h49m58.763644651s) [upload=278145]: Done uploading segments... Nov 22 11:54:00.212200729 (315h49m58.763671191s) [upload=278145]: Upload finished. err: <nil> Duration: 5.776151462s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.690285217s trace: https://jaeger.dev.storj.io/trace/76f1ecbff69b5d4e FAIL (2025-11-22 12:18:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, Piece ID: I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ) context deadline exceeded; piecestore: (Node ID: 12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, Piece ID: YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA) context deadline exceeded; piecestore: (Node ID: 1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, Piece ID: G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ) context deadline exceeded; piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA) context deadline exceeded ======LOGS====== nts... Nov 22 11:53:53.345386809 (315h49m51.89685727s) [upload=278145]: Got next segment. Inline: false Nov 22 11:53:53.345428137 (315h49m51.896898598s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 11:53:53.345433948 (315h49m51.896904419s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 11:53:54.603541081 (315h49m53.155011552s) [seg_pos={0 0}, upload=278145]: Begin upload segment... Nov 22 11:53:54.603595445 (315h49m53.155065916s) [seg_pos={0 0}, upload=278145]: Done begin upload segment. Nov 22 11:53:54.603668043 (315h49m53.155138494s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.603684554 (315h49m53.155155005s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604505828 (315h49m53.155976289s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604973038 (315h49m53.156443499s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.605425900 (315h49m53.156896361s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606323339 (315h49m53.15779381s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606878326 (315h49m53.158348777s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.607588267 (315h49m53.159058728s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610211763 (315h49m53.161682215s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610336751 (315h49m53.161807212s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612196994 (315h49m53.163667455s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612485593 (315h49m53.163956064s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614299567 (315h49m53.165770028s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614454202 (315h49m53.165924673s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.616397682 (315h49m53.167868143s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.621720077 (315h49m53.173190538s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:56.445932556 (315h49m54.997403007s) [upload=278145]: Next returned nil segment Nov 22 11:53:56.445949048 (315h49m54.997419509s) [upload=278145]: Waiting for error group managing segments... Nov 22 11:53:56.734588580 (315h49m55.286059041s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.796169791 (315h49m55.347640252s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.800552848 (315h49m55.352023309s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.802134119 (315h49m55.35360458s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.811866140 (315h49m55.363336591s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.820962632 (315h49m55.372433093s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.853755886 (315h49m55.405226347s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.857215724 (315h49m55.408686185s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.067648917 (315h49m55.619119378s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.093474313 (315h49m55.644944774s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.104509907 (315h49m55.655980378s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.505088550 (315h49m56.056559011s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609615419 (315h49m56.16108589s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609676946 (315h49m56.161147437s) [seg_pos={0 0}, upload=278145]: Segment reached optimal threshold of 13 pieces. Nov 22 11:53:57.609972077 (315h49m56.161442539s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection Nov 22 11:53:57.609982889 (315h49m56.16145336s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610025941 (315h49m56.161496432s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 22 11:53:57.610037101 (315h49m56.161507553s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610213978 (315h49m56.161684439s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection Nov 22 11:53:57.610224649 (315h49m56.16169511s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610261058 (315h49m56.16173152s) [seg_pos={0 0}, upload=278145]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 11:53:57.610421113 (315h49m56.161891584s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 11:53:57.610430852 (315h49m56.161901313s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 11:54:00.212174190 (315h49m58.763644651s) [upload=278145]: Done uploading segments... Nov 22 11:54:00.212200729 (315h49m58.763671191s) [upload=278145]: Upload finished. err: <nil> Duration: 5.69030713s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.694603421s trace: https://jaeger.dev.storj.io/trace/76f1ecbff69b5d4e FAIL (2025-11-22 12:18:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, Piece ID: I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ) context deadline exceeded; piecestore: (Node ID: 12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, Piece ID: YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA) context deadline exceeded; piecestore: (Node ID: 1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, Piece ID: G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ) context deadline exceeded; piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA) context deadline exceeded ======LOGS====== nts... Nov 22 11:53:53.345386809 (315h49m51.89685727s) [upload=278145]: Got next segment. Inline: false Nov 22 11:53:53.345428137 (315h49m51.896898598s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 11:53:53.345433948 (315h49m51.896904419s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 11:53:54.603541081 (315h49m53.155011552s) [seg_pos={0 0}, upload=278145]: Begin upload segment... Nov 22 11:53:54.603595445 (315h49m53.155065916s) [seg_pos={0 0}, upload=278145]: Done begin upload segment. Nov 22 11:53:54.603668043 (315h49m53.155138494s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.603684554 (315h49m53.155155005s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604505828 (315h49m53.155976289s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604973038 (315h49m53.156443499s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.605425900 (315h49m53.156896361s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606323339 (315h49m53.15779381s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606878326 (315h49m53.158348777s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.607588267 (315h49m53.159058728s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610211763 (315h49m53.161682215s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610336751 (315h49m53.161807212s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612196994 (315h49m53.163667455s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612485593 (315h49m53.163956064s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614299567 (315h49m53.165770028s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614454202 (315h49m53.165924673s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.616397682 (315h49m53.167868143s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.621720077 (315h49m53.173190538s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:56.445932556 (315h49m54.997403007s) [upload=278145]: Next returned nil segment Nov 22 11:53:56.445949048 (315h49m54.997419509s) [upload=278145]: Waiting for error group managing segments... Nov 22 11:53:56.734588580 (315h49m55.286059041s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.796169791 (315h49m55.347640252s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.800552848 (315h49m55.352023309s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.802134119 (315h49m55.35360458s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.811866140 (315h49m55.363336591s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.820962632 (315h49m55.372433093s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.853755886 (315h49m55.405226347s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.857215724 (315h49m55.408686185s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.067648917 (315h49m55.619119378s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.093474313 (315h49m55.644944774s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.104509907 (315h49m55.655980378s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.505088550 (315h49m56.056559011s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609615419 (315h49m56.16108589s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609676946 (315h49m56.161147437s) [seg_pos={0 0}, upload=278145]: Segment reached optimal threshold of 13 pieces. Nov 22 11:53:57.609972077 (315h49m56.161442539s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection Nov 22 11:53:57.609982889 (315h49m56.16145336s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610025941 (315h49m56.161496432s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 22 11:53:57.610037101 (315h49m56.161507553s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610213978 (315h49m56.161684439s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection Nov 22 11:53:57.610224649 (315h49m56.16169511s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610261058 (315h49m56.16173152s) [seg_pos={0 0}, upload=278145]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 11:53:57.610421113 (315h49m56.161891584s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 11:53:57.610430852 (315h49m56.161901313s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 11:54:00.212174190 (315h49m58.763644651s) [upload=278145]: Done uploading segments... Nov 22 11:54:00.212200729 (315h49m58.763671191s) [upload=278145]: Upload finished. err: <nil> Duration: 5.694696439s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................X....................................] duration: 3.529896948s trace: https://jaeger.dev.storj.io/trace/469b06ffc0b434d4 FAIL (2025-11-22 12:17:46 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.529928648s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.719399064s trace: https://jaeger.dev.storj.io/trace/76f1ecbff69b5d4e FAIL (2025-11-22 12:17:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, Piece ID: I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ) context deadline exceeded; piecestore: (Node ID: 12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, Piece ID: YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA) context deadline exceeded; piecestore: (Node ID: 1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, Piece ID: G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ) context deadline exceeded; piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA) context deadline exceeded ======LOGS====== nts... Nov 22 11:53:53.345386809 (315h49m51.89685727s) [upload=278145]: Got next segment. Inline: false Nov 22 11:53:53.345428137 (315h49m51.896898598s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 11:53:53.345433948 (315h49m51.896904419s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 11:53:54.603541081 (315h49m53.155011552s) [seg_pos={0 0}, upload=278145]: Begin upload segment... Nov 22 11:53:54.603595445 (315h49m53.155065916s) [seg_pos={0 0}, upload=278145]: Done begin upload segment. Nov 22 11:53:54.603668043 (315h49m53.155138494s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.603684554 (315h49m53.155155005s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604505828 (315h49m53.155976289s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604973038 (315h49m53.156443499s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.605425900 (315h49m53.156896361s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606323339 (315h49m53.15779381s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606878326 (315h49m53.158348777s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.607588267 (315h49m53.159058728s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610211763 (315h49m53.161682215s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610336751 (315h49m53.161807212s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612196994 (315h49m53.163667455s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612485593 (315h49m53.163956064s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614299567 (315h49m53.165770028s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614454202 (315h49m53.165924673s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.616397682 (315h49m53.167868143s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.621720077 (315h49m53.173190538s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:56.445932556 (315h49m54.997403007s) [upload=278145]: Next returned nil segment Nov 22 11:53:56.445949048 (315h49m54.997419509s) [upload=278145]: Waiting for error group managing segments... Nov 22 11:53:56.734588580 (315h49m55.286059041s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.796169791 (315h49m55.347640252s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.800552848 (315h49m55.352023309s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.802134119 (315h49m55.35360458s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.811866140 (315h49m55.363336591s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.820962632 (315h49m55.372433093s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.853755886 (315h49m55.405226347s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.857215724 (315h49m55.408686185s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.067648917 (315h49m55.619119378s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.093474313 (315h49m55.644944774s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.104509907 (315h49m55.655980378s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.505088550 (315h49m56.056559011s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609615419 (315h49m56.16108589s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609676946 (315h49m56.161147437s) [seg_pos={0 0}, upload=278145]: Segment reached optimal threshold of 13 pieces. Nov 22 11:53:57.609972077 (315h49m56.161442539s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection Nov 22 11:53:57.609982889 (315h49m56.16145336s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610025941 (315h49m56.161496432s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 22 11:53:57.610037101 (315h49m56.161507553s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610213978 (315h49m56.161684439s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection Nov 22 11:53:57.610224649 (315h49m56.16169511s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610261058 (315h49m56.16173152s) [seg_pos={0 0}, upload=278145]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 11:53:57.610421113 (315h49m56.161891584s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 11:53:57.610430852 (315h49m56.161901313s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 11:54:00.212174190 (315h49m58.763644651s) [upload=278145]: Done uploading segments... Nov 22 11:54:00.212200729 (315h49m58.763671191s) [upload=278145]: Upload finished. err: <nil> Duration: 5.719445673s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.72922765s trace: https://jaeger.dev.storj.io/trace/76f1ecbff69b5d4e FAIL (2025-11-22 12:16:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, Piece ID: I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ) context deadline exceeded; piecestore: (Node ID: 12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, Piece ID: YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA) context deadline exceeded; piecestore: (Node ID: 1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, Piece ID: G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ) context deadline exceeded; piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA) context deadline exceeded ======LOGS====== nts... Nov 22 11:53:53.345386809 (315h49m51.89685727s) [upload=278145]: Got next segment. Inline: false Nov 22 11:53:53.345428137 (315h49m51.896898598s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 11:53:53.345433948 (315h49m51.896904419s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 11:53:54.603541081 (315h49m53.155011552s) [seg_pos={0 0}, upload=278145]: Begin upload segment... Nov 22 11:53:54.603595445 (315h49m53.155065916s) [seg_pos={0 0}, upload=278145]: Done begin upload segment. Nov 22 11:53:54.603668043 (315h49m53.155138494s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.603684554 (315h49m53.155155005s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604505828 (315h49m53.155976289s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604973038 (315h49m53.156443499s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.605425900 (315h49m53.156896361s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606323339 (315h49m53.15779381s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606878326 (315h49m53.158348777s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.607588267 (315h49m53.159058728s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610211763 (315h49m53.161682215s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610336751 (315h49m53.161807212s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612196994 (315h49m53.163667455s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612485593 (315h49m53.163956064s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614299567 (315h49m53.165770028s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614454202 (315h49m53.165924673s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.616397682 (315h49m53.167868143s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.621720077 (315h49m53.173190538s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:56.445932556 (315h49m54.997403007s) [upload=278145]: Next returned nil segment Nov 22 11:53:56.445949048 (315h49m54.997419509s) [upload=278145]: Waiting for error group managing segments... Nov 22 11:53:56.734588580 (315h49m55.286059041s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.796169791 (315h49m55.347640252s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.800552848 (315h49m55.352023309s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.802134119 (315h49m55.35360458s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.811866140 (315h49m55.363336591s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.820962632 (315h49m55.372433093s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.853755886 (315h49m55.405226347s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.857215724 (315h49m55.408686185s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.067648917 (315h49m55.619119378s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.093474313 (315h49m55.644944774s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.104509907 (315h49m55.655980378s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.505088550 (315h49m56.056559011s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609615419 (315h49m56.16108589s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609676946 (315h49m56.161147437s) [seg_pos={0 0}, upload=278145]: Segment reached optimal threshold of 13 pieces. Nov 22 11:53:57.609972077 (315h49m56.161442539s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection Nov 22 11:53:57.609982889 (315h49m56.16145336s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610025941 (315h49m56.161496432s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 22 11:53:57.610037101 (315h49m56.161507553s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610213978 (315h49m56.161684439s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection Nov 22 11:53:57.610224649 (315h49m56.16169511s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610261058 (315h49m56.16173152s) [seg_pos={0 0}, upload=278145]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 11:53:57.610421113 (315h49m56.161891584s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 11:53:57.610430852 (315h49m56.161901313s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 11:54:00.212174190 (315h49m58.763644651s) [upload=278145]: Done uploading segments... Nov 22 11:54:00.212200729 (315h49m58.763671191s) [upload=278145]: Upload finished. err: <nil> Duration: 5.729256806s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................X...................................] duration: 3.388227763s trace: https://jaeger.dev.storj.io/trace/469b06ffc0b434d4 FAIL (2025-11-22 12:16:38 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.388264433s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.751883523s trace: https://jaeger.dev.storj.io/trace/76f1ecbff69b5d4e FAIL (2025-11-22 12:16:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, Piece ID: I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ) context deadline exceeded; piecestore: (Node ID: 12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, Piece ID: YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA) context deadline exceeded; piecestore: (Node ID: 1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, Piece ID: G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ) context deadline exceeded; piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA) context deadline exceeded ======LOGS====== nts... Nov 22 11:53:53.345386809 (315h49m51.89685727s) [upload=278145]: Got next segment. Inline: false Nov 22 11:53:53.345428137 (315h49m51.896898598s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 11:53:53.345433948 (315h49m51.896904419s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 11:53:54.603541081 (315h49m53.155011552s) [seg_pos={0 0}, upload=278145]: Begin upload segment... Nov 22 11:53:54.603595445 (315h49m53.155065916s) [seg_pos={0 0}, upload=278145]: Done begin upload segment. Nov 22 11:53:54.603668043 (315h49m53.155138494s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.603684554 (315h49m53.155155005s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604505828 (315h49m53.155976289s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604973038 (315h49m53.156443499s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.605425900 (315h49m53.156896361s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606323339 (315h49m53.15779381s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606878326 (315h49m53.158348777s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.607588267 (315h49m53.159058728s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610211763 (315h49m53.161682215s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610336751 (315h49m53.161807212s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612196994 (315h49m53.163667455s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612485593 (315h49m53.163956064s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614299567 (315h49m53.165770028s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614454202 (315h49m53.165924673s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.616397682 (315h49m53.167868143s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.621720077 (315h49m53.173190538s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:56.445932556 (315h49m54.997403007s) [upload=278145]: Next returned nil segment Nov 22 11:53:56.445949048 (315h49m54.997419509s) [upload=278145]: Waiting for error group managing segments... Nov 22 11:53:56.734588580 (315h49m55.286059041s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.796169791 (315h49m55.347640252s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.800552848 (315h49m55.352023309s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.802134119 (315h49m55.35360458s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.811866140 (315h49m55.363336591s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.820962632 (315h49m55.372433093s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.853755886 (315h49m55.405226347s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.857215724 (315h49m55.408686185s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.067648917 (315h49m55.619119378s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.093474313 (315h49m55.644944774s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.104509907 (315h49m55.655980378s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.505088550 (315h49m56.056559011s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609615419 (315h49m56.16108589s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609676946 (315h49m56.161147437s) [seg_pos={0 0}, upload=278145]: Segment reached optimal threshold of 13 pieces. Nov 22 11:53:57.609972077 (315h49m56.161442539s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection Nov 22 11:53:57.609982889 (315h49m56.16145336s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610025941 (315h49m56.161496432s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 22 11:53:57.610037101 (315h49m56.161507553s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610213978 (315h49m56.161684439s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection Nov 22 11:53:57.610224649 (315h49m56.16169511s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610261058 (315h49m56.16173152s) [seg_pos={0 0}, upload=278145]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 11:53:57.610421113 (315h49m56.161891584s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 11:53:57.610430852 (315h49m56.161901313s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 11:54:00.212174190 (315h49m58.763644651s) [upload=278145]: Done uploading segments... Nov 22 11:54:00.212200729 (315h49m58.763671191s) [upload=278145]: Upload finished. err: <nil> Duration: 5.75191297s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................X..................................] duration: 3.545552903s trace: https://jaeger.dev.storj.io/trace/469b06ffc0b434d4 FAIL (2025-11-22 12:15:27 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.545577981s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.648598339s trace: https://jaeger.dev.storj.io/trace/76f1ecbff69b5d4e FAIL (2025-11-22 12:14:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, Piece ID: I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ) context deadline exceeded; piecestore: (Node ID: 12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, Piece ID: YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA) context deadline exceeded; piecestore: (Node ID: 1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, Piece ID: G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ) context deadline exceeded; piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA) context deadline exceeded ======LOGS====== nts... Nov 22 11:53:53.345386809 (315h49m51.89685727s) [upload=278145]: Got next segment. Inline: false Nov 22 11:53:53.345428137 (315h49m51.896898598s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 11:53:53.345433948 (315h49m51.896904419s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 11:53:54.603541081 (315h49m53.155011552s) [seg_pos={0 0}, upload=278145]: Begin upload segment... Nov 22 11:53:54.603595445 (315h49m53.155065916s) [seg_pos={0 0}, upload=278145]: Done begin upload segment. Nov 22 11:53:54.603668043 (315h49m53.155138494s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.603684554 (315h49m53.155155005s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604505828 (315h49m53.155976289s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604973038 (315h49m53.156443499s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.605425900 (315h49m53.156896361s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606323339 (315h49m53.15779381s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606878326 (315h49m53.158348777s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.607588267 (315h49m53.159058728s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610211763 (315h49m53.161682215s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610336751 (315h49m53.161807212s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612196994 (315h49m53.163667455s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612485593 (315h49m53.163956064s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614299567 (315h49m53.165770028s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614454202 (315h49m53.165924673s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.616397682 (315h49m53.167868143s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.621720077 (315h49m53.173190538s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:56.445932556 (315h49m54.997403007s) [upload=278145]: Next returned nil segment Nov 22 11:53:56.445949048 (315h49m54.997419509s) [upload=278145]: Waiting for error group managing segments... Nov 22 11:53:56.734588580 (315h49m55.286059041s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.796169791 (315h49m55.347640252s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.800552848 (315h49m55.352023309s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.802134119 (315h49m55.35360458s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.811866140 (315h49m55.363336591s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.820962632 (315h49m55.372433093s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.853755886 (315h49m55.405226347s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.857215724 (315h49m55.408686185s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.067648917 (315h49m55.619119378s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.093474313 (315h49m55.644944774s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.104509907 (315h49m55.655980378s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.505088550 (315h49m56.056559011s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609615419 (315h49m56.16108589s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609676946 (315h49m56.161147437s) [seg_pos={0 0}, upload=278145]: Segment reached optimal threshold of 13 pieces. Nov 22 11:53:57.609972077 (315h49m56.161442539s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection Nov 22 11:53:57.609982889 (315h49m56.16145336s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610025941 (315h49m56.161496432s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 22 11:53:57.610037101 (315h49m56.161507553s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610213978 (315h49m56.161684439s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection Nov 22 11:53:57.610224649 (315h49m56.16169511s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610261058 (315h49m56.16173152s) [seg_pos={0 0}, upload=278145]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 11:53:57.610421113 (315h49m56.161891584s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 11:53:57.610430852 (315h49m56.161901313s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 11:54:00.212174190 (315h49m58.763644651s) [upload=278145]: Done uploading segments... Nov 22 11:54:00.212200729 (315h49m58.763671191s) [upload=278145]: Upload finished. err: <nil> Duration: 5.648629377s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................X.................................] duration: 3.557826465s trace: https://jaeger.dev.storj.io/trace/469b06ffc0b434d4 FAIL (2025-11-22 12:14:47 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.557853818s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................X................................] duration: 3.529121084s trace: https://jaeger.dev.storj.io/trace/469b06ffc0b434d4 FAIL (2025-11-22 12:14:21 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.529159336s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.730858518s trace: https://jaeger.dev.storj.io/trace/76f1ecbff69b5d4e FAIL (2025-11-22 12:13:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, Piece ID: I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ) context deadline exceeded; piecestore: (Node ID: 12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, Piece ID: YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA) context deadline exceeded; piecestore: (Node ID: 1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, Piece ID: G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ) context deadline exceeded; piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA) context deadline exceeded ======LOGS====== nts... Nov 22 11:53:53.345386809 (315h49m51.89685727s) [upload=278145]: Got next segment. Inline: false Nov 22 11:53:53.345428137 (315h49m51.896898598s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 11:53:53.345433948 (315h49m51.896904419s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 11:53:54.603541081 (315h49m53.155011552s) [seg_pos={0 0}, upload=278145]: Begin upload segment... Nov 22 11:53:54.603595445 (315h49m53.155065916s) [seg_pos={0 0}, upload=278145]: Done begin upload segment. Nov 22 11:53:54.603668043 (315h49m53.155138494s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.603684554 (315h49m53.155155005s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604505828 (315h49m53.155976289s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604973038 (315h49m53.156443499s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.605425900 (315h49m53.156896361s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606323339 (315h49m53.15779381s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606878326 (315h49m53.158348777s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.607588267 (315h49m53.159058728s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610211763 (315h49m53.161682215s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610336751 (315h49m53.161807212s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612196994 (315h49m53.163667455s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612485593 (315h49m53.163956064s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614299567 (315h49m53.165770028s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614454202 (315h49m53.165924673s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.616397682 (315h49m53.167868143s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.621720077 (315h49m53.173190538s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:56.445932556 (315h49m54.997403007s) [upload=278145]: Next returned nil segment Nov 22 11:53:56.445949048 (315h49m54.997419509s) [upload=278145]: Waiting for error group managing segments... Nov 22 11:53:56.734588580 (315h49m55.286059041s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.796169791 (315h49m55.347640252s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.800552848 (315h49m55.352023309s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.802134119 (315h49m55.35360458s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.811866140 (315h49m55.363336591s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.820962632 (315h49m55.372433093s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.853755886 (315h49m55.405226347s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.857215724 (315h49m55.408686185s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.067648917 (315h49m55.619119378s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.093474313 (315h49m55.644944774s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.104509907 (315h49m55.655980378s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.505088550 (315h49m56.056559011s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609615419 (315h49m56.16108589s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609676946 (315h49m56.161147437s) [seg_pos={0 0}, upload=278145]: Segment reached optimal threshold of 13 pieces. Nov 22 11:53:57.609972077 (315h49m56.161442539s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection Nov 22 11:53:57.609982889 (315h49m56.16145336s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610025941 (315h49m56.161496432s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 22 11:53:57.610037101 (315h49m56.161507553s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610213978 (315h49m56.161684439s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection Nov 22 11:53:57.610224649 (315h49m56.16169511s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610261058 (315h49m56.16173152s) [seg_pos={0 0}, upload=278145]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 11:53:57.610421113 (315h49m56.161891584s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 11:53:57.610430852 (315h49m56.161901313s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 11:54:00.212174190 (315h49m58.763644651s) [upload=278145]: Done uploading segments... Nov 22 11:54:00.212200729 (315h49m58.763671191s) [upload=278145]: Upload finished. err: <nil> Duration: 5.730889666s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.717209297s trace: https://jaeger.dev.storj.io/trace/76f1ecbff69b5d4e FAIL (2025-11-22 12:13:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, Piece ID: I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ) context deadline exceeded; piecestore: (Node ID: 12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, Piece ID: YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA) context deadline exceeded; piecestore: (Node ID: 1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, Piece ID: G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ) context deadline exceeded; piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA) context deadline exceeded ======LOGS====== nts... Nov 22 11:53:53.345386809 (315h49m51.89685727s) [upload=278145]: Got next segment. Inline: false Nov 22 11:53:53.345428137 (315h49m51.896898598s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 11:53:53.345433948 (315h49m51.896904419s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 11:53:54.603541081 (315h49m53.155011552s) [seg_pos={0 0}, upload=278145]: Begin upload segment... Nov 22 11:53:54.603595445 (315h49m53.155065916s) [seg_pos={0 0}, upload=278145]: Done begin upload segment. Nov 22 11:53:54.603668043 (315h49m53.155138494s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.603684554 (315h49m53.155155005s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604505828 (315h49m53.155976289s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604973038 (315h49m53.156443499s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.605425900 (315h49m53.156896361s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606323339 (315h49m53.15779381s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606878326 (315h49m53.158348777s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.607588267 (315h49m53.159058728s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610211763 (315h49m53.161682215s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610336751 (315h49m53.161807212s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612196994 (315h49m53.163667455s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612485593 (315h49m53.163956064s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614299567 (315h49m53.165770028s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614454202 (315h49m53.165924673s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.616397682 (315h49m53.167868143s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.621720077 (315h49m53.173190538s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:56.445932556 (315h49m54.997403007s) [upload=278145]: Next returned nil segment Nov 22 11:53:56.445949048 (315h49m54.997419509s) [upload=278145]: Waiting for error group managing segments... Nov 22 11:53:56.734588580 (315h49m55.286059041s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.796169791 (315h49m55.347640252s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.800552848 (315h49m55.352023309s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.802134119 (315h49m55.35360458s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.811866140 (315h49m55.363336591s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.820962632 (315h49m55.372433093s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.853755886 (315h49m55.405226347s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.857215724 (315h49m55.408686185s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.067648917 (315h49m55.619119378s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.093474313 (315h49m55.644944774s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.104509907 (315h49m55.655980378s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.505088550 (315h49m56.056559011s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609615419 (315h49m56.16108589s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609676946 (315h49m56.161147437s) [seg_pos={0 0}, upload=278145]: Segment reached optimal threshold of 13 pieces. Nov 22 11:53:57.609972077 (315h49m56.161442539s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection Nov 22 11:53:57.609982889 (315h49m56.16145336s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610025941 (315h49m56.161496432s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 22 11:53:57.610037101 (315h49m56.161507553s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610213978 (315h49m56.161684439s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection Nov 22 11:53:57.610224649 (315h49m56.16169511s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610261058 (315h49m56.16173152s) [seg_pos={0 0}, upload=278145]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 11:53:57.610421113 (315h49m56.161891584s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 11:53:57.610430852 (315h49m56.161901313s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 11:54:00.212174190 (315h49m58.763644651s) [upload=278145]: Done uploading segments... Nov 22 11:54:00.212200729 (315h49m58.763671191s) [upload=278145]: Upload finished. err: <nil> Duration: 5.717236178s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................X...............................] duration: 3.506898849s trace: https://jaeger.dev.storj.io/trace/469b06ffc0b434d4 FAIL (2025-11-22 12:13:08 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.506931691s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.790513587s trace: https://jaeger.dev.storj.io/trace/76f1ecbff69b5d4e FAIL (2025-11-22 12:12:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, Piece ID: I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ) context deadline exceeded; piecestore: (Node ID: 12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, Piece ID: YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA) context deadline exceeded; piecestore: (Node ID: 1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, Piece ID: G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ) context deadline exceeded; piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA) context deadline exceeded ======LOGS====== nts... Nov 22 11:53:53.345386809 (315h49m51.89685727s) [upload=278145]: Got next segment. Inline: false Nov 22 11:53:53.345428137 (315h49m51.896898598s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 11:53:53.345433948 (315h49m51.896904419s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 11:53:54.603541081 (315h49m53.155011552s) [seg_pos={0 0}, upload=278145]: Begin upload segment... Nov 22 11:53:54.603595445 (315h49m53.155065916s) [seg_pos={0 0}, upload=278145]: Done begin upload segment. Nov 22 11:53:54.603668043 (315h49m53.155138494s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.603684554 (315h49m53.155155005s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604505828 (315h49m53.155976289s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604973038 (315h49m53.156443499s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.605425900 (315h49m53.156896361s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606323339 (315h49m53.15779381s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606878326 (315h49m53.158348777s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.607588267 (315h49m53.159058728s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610211763 (315h49m53.161682215s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610336751 (315h49m53.161807212s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612196994 (315h49m53.163667455s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612485593 (315h49m53.163956064s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614299567 (315h49m53.165770028s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614454202 (315h49m53.165924673s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.616397682 (315h49m53.167868143s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.621720077 (315h49m53.173190538s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:56.445932556 (315h49m54.997403007s) [upload=278145]: Next returned nil segment Nov 22 11:53:56.445949048 (315h49m54.997419509s) [upload=278145]: Waiting for error group managing segments... Nov 22 11:53:56.734588580 (315h49m55.286059041s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.796169791 (315h49m55.347640252s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.800552848 (315h49m55.352023309s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.802134119 (315h49m55.35360458s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.811866140 (315h49m55.363336591s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.820962632 (315h49m55.372433093s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.853755886 (315h49m55.405226347s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.857215724 (315h49m55.408686185s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.067648917 (315h49m55.619119378s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.093474313 (315h49m55.644944774s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.104509907 (315h49m55.655980378s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.505088550 (315h49m56.056559011s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609615419 (315h49m56.16108589s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609676946 (315h49m56.161147437s) [seg_pos={0 0}, upload=278145]: Segment reached optimal threshold of 13 pieces. Nov 22 11:53:57.609972077 (315h49m56.161442539s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection Nov 22 11:53:57.609982889 (315h49m56.16145336s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610025941 (315h49m56.161496432s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 22 11:53:57.610037101 (315h49m56.161507553s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610213978 (315h49m56.161684439s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection Nov 22 11:53:57.610224649 (315h49m56.16169511s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610261058 (315h49m56.16173152s) [seg_pos={0 0}, upload=278145]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 11:53:57.610421113 (315h49m56.161891584s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 11:53:57.610430852 (315h49m56.161901313s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 11:54:00.212174190 (315h49m58.763644651s) [upload=278145]: Done uploading segments... Nov 22 11:54:00.212200729 (315h49m58.763671191s) [upload=278145]: Upload finished. err: <nil> Duration: 5.790560147s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................X..............................] duration: 3.376209031s trace: https://jaeger.dev.storj.io/trace/469b06ffc0b434d4 FAIL (2025-11-22 12:12:00 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.37622976s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.835015648s trace: https://jaeger.dev.storj.io/trace/76f1ecbff69b5d4e FAIL (2025-11-22 12:11:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, Piece ID: I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ) context deadline exceeded; piecestore: (Node ID: 12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, Piece ID: YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A) context deadline exceeded; piecestore: (Node ID: 12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, Piece ID: P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA) context deadline exceeded; piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ) context deadline exceeded; piecestore: (Node ID: 1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, Piece ID: ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA) context deadline exceeded; piecestore: (Node ID: 1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, Piece ID: G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ) context deadline exceeded; piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA) context deadline exceeded ======LOGS====== nts... Nov 22 11:53:53.345386809 (315h49m51.89685727s) [upload=278145]: Got next segment. Inline: false Nov 22 11:53:53.345428137 (315h49m51.896898598s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 11:53:53.345433948 (315h49m51.896904419s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 11:53:54.603541081 (315h49m53.155011552s) [seg_pos={0 0}, upload=278145]: Begin upload segment... Nov 22 11:53:54.603595445 (315h49m53.155065916s) [seg_pos={0 0}, upload=278145]: Done begin upload segment. Nov 22 11:53:54.603668043 (315h49m53.155138494s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.603684554 (315h49m53.155155005s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604505828 (315h49m53.155976289s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.604973038 (315h49m53.156443499s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.605425900 (315h49m53.156896361s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606323339 (315h49m53.15779381s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.606878326 (315h49m53.158348777s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.607588267 (315h49m53.159058728s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610211763 (315h49m53.161682215s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.610336751 (315h49m53.161807212s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612196994 (315h49m53.163667455s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.612485593 (315h49m53.163956064s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614299567 (315h49m53.165770028s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.614454202 (315h49m53.165924673s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.616397682 (315h49m53.167868143s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:54.621720077 (315h49m53.173190538s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Uploading piece... Nov 22 11:53:56.445932556 (315h49m54.997403007s) [upload=278145]: Next returned nil segment Nov 22 11:53:56.445949048 (315h49m54.997419509s) [upload=278145]: Waiting for error group managing segments... Nov 22 11:53:56.734588580 (315h49m55.286059041s) [noise=true , address=77.90.31.10:4517 , piece_id=KLXWC5RY274YKMLT2U5SBPCNQKIRXDOTPQ7MD76HXA3F2VCYRIEA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.796169791 (315h49m55.347640252s) [noise=true , address=145.79.227.20:4438 , piece_id=NR272LQGDUPLQHUXRKLHYME3GC5XJP3XKPGZGNJTQPEI2DHB4EOA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.800552848 (315h49m55.352023309s) [noise=true , address=45.91.116.11:4119 , piece_id=FGNP4Z5IL65VGJ7NMOJK5JW4Y644RFZG6MHWUOCLFKBJ26KF5W5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.802134119 (315h49m55.35360458s) [noise=true , address=145.79.227.4:4187 , piece_id=P7IQ6DZWR4DIUQNLGPH43R4A4JIUSEWSXLPNY75TZCL3EFFIVVRA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.811866140 (315h49m55.363336591s) [noise=true , address=145.79.227.17:4969 , piece_id=I2NZAJH2UWZW6MKLZ2G57PRIOV7B3AXZFK73L35EZ5O2TPIJUYGQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.820962632 (315h49m55.372433093s) [noise=true , address=77.90.31.15:4698 , piece_id=3Z7MYVQ62TX4BKJASUPTJTWDZH57PEVBSPT6OV2SMMZMTG2KCYZQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.853755886 (315h49m55.405226347s) [noise=true , address=145.79.227.21:4474 , piece_id=HYSRVRV3ALPITM7ZJ2GHPIGLTZ54YH4KYUZ36PTWAU6V2XBLURKQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:56.857215724 (315h49m55.408686185s) [noise=true , address=145.79.227.14:4837 , piece_id=TCYBUWS4YPUVLJ6QAPX2QIRT7O2HSBVEKVPPKHCTK336NNNPO7PQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.067648917 (315h49m55.619119378s) [noise=true , address=145.79.227.25:4619 , piece_id=ETABGHSUERS3TQGO7W36TBWBRS2PLMWYMDNRO4FWXY4OFIZTCHTA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.093474313 (315h49m55.644944774s) [noise=true , address=145.79.227.9:4366 , piece_id=G45HAZUXAQ356UXTDCVWUG5RLSDUSFOEDZHRFNYAHEDAHZURXHYQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.104509907 (315h49m55.655980378s) [noise=true , address=145.79.227.12:4767 , piece_id=FSJE5Q3Z7KHDGYUMIWRXJKTDOT5QFHZU7WQ4SHC3VQGGWJRNVEOQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.505088550 (315h49m56.056559011s) [noise=true , address=145.79.227.31:4799 , piece_id=RIAYMYQN4MN444NKV5N6VVNNHFLU5LRALEWGDWIWHOEKF33EULTQ, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609615419 (315h49m56.16108589s) [noise=true , address=145.79.227.28:4726 , piece_id=YZNVUZGYX2OCQODI6E7X2OH3VGJ2OAWROC5FHB75QPU5URINKV5A, seg_pos={0 0}, upload=278145]: Done uploading piece. err: <nil> Nov 22 11:53:57.609676946 (315h49m56.161147437s) [seg_pos={0 0}, upload=278145]: Segment reached optimal threshold of 13 pieces. Nov 22 11:53:57.609972077 (315h49m56.161442539s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46136->145.79.227.8:4330: use of closed network connection Nov 22 11:53:57.609982889 (315h49m56.16145336s) [noise=true , address=145.79.227.8:4330 , piece_id=AC3YGQI3BRZ7ZSZ46GNXTCBB3PCVYYDE34WVGSLYSDHXNHN45WMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610025941 (315h49m56.161496432s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: expected piece hash; context canceled; EOF Nov 22 11:53:57.610037101 (315h49m56.161507553s) [noise=true , address=145.79.227.32:4834 , piece_id=U3CNGFFXT25TXDFI56VDND5ENH5QBS4RYP5KH6DKOYZBMGAGVIJA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610213978 (315h49m56.161684439s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54808->145.79.227.24:4583: use of closed network connection Nov 22 11:53:57.610224649 (315h49m56.16169511s) [noise=true , address=145.79.227.24:4583 , piece_id=CDF5LKYTRHKN474ETIOWHYZKJWF76F2DNFACIJJL45PR5VVAKUMA, seg_pos={0 0}, upload=278145]: Long tail cancellation due to optimal threshold Nov 22 11:53:57.610261058 (315h49m56.16173152s) [seg_pos={0 0}, upload=278145]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 22 11:53:57.610421113 (315h49m56.161891584s) [upload=278145]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 22 11:53:57.610430852 (315h49m56.161901313s) [upload=278145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 22 11:54:00.212174190 (315h49m58.763644651s) [upload=278145]: Done uploading segments... Nov 22 11:54:00.212200729 (315h49m58.763671191s) [upload=278145]: Upload finished. err: <nil> Duration: 5.835049022s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)