Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 348383 reports (Page 183 of 6968)
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
54 out of 60 is failed (threshold=6) [X.XX.XXXXXXX.XX.XXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXX]
duration: 
5.00105216s
trace: 
https://jaeger.dev.storj.io/trace/49ad2cd4c048aeeb
FAIL (2025-12-29 13:41:59 UTC) 
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001076837s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)
antiblip: 
1 out of 60 is failed (threshold=3) [.........................................................X..]
duration: 
19.802423807s
trace: 
https://jaeger.dev.storj.io/trace/1df9ff77465ee3c2
FAIL (2025-12-29 13:41:42 UTC) 
orbiter warning: context deadline exceeded
Duration: 19.802447964s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
7 out of 60 is failed (threshold=3) [....X....X.X....X.X....................................X...X]
duration: 
30.001614619s
trace: 
https://jaeger.dev.storj.io/trace/1e89d4bbf5f848b4
FAIL (2025-12-29 13:41:40 UTC) 
orbiter warning: context deadline exceeded
Duration: 30.00163087s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
31 out of 60 is failed (threshold=3) [.XXX.X.X.X.XX..X.XXX.XX..X.XX..XXX...X..X...X...X.XXXX.X.XX.]
duration: 
6.908379569s
trace: 
https://jaeger.dev.storj.io/trace/11c1b2787abeda0a
FAIL (2025-12-29 13:41:34 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
segments...
Dec 29 13:40:33.713978579 (574h11m33.585061624s) [upload=503383]: Got next segment. Inline: false
Dec 29 13:40:33.714036390 (574h11m33.585119435s) [upload=503383]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 13:40:33.714041389 (574h11m33.585124424s) [upload=503383]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 13:40:33.960974115 (574h11m33.83205715s) [seg_pos={0 0}, upload=503383]: Begin upload segment...
Dec 29 13:40:33.961059237 (574h11m33.832142302s) [noise=true , address=145.79.227.2:4115    , piece_id=WVKUZECFJ24LFZTDJOMBFTLYDNRXD6M3YHRLKCPFIYL6MDE2FGWA, seg_pos={0 0}, upload=503383]: Uploading piece...
Dec 29 13:40:33.961114292 (574h11m33.832197337s) [seg_pos={0 0}, upload=503383]: Done begin upload segment.
Dec 29 13:40:33.961256663 (574h11m33.832339708s) [noise=true , address=145.79.227.39:4014   , piece_id=L3XYIRTFGKKLGQ5DD33GIIBNUHPI4E63V2JWFMRQ2ZX2OQJ6LGZQ, seg_pos={0 0}, upload=503383]: Uploading piece...
Dec 29 13:40:33.961769740 (574h11m33.832852775s) [noise=true , address=145.79.227.3:4149    , piece_id=S455YAPDXPDXBIQERWZSWKRD5XMZTKICA5JFTYNTGPT6BPTTM5PQ, seg_pos={0 0}, upload=503383]: Uploading piece...
Dec 29 13:40:33.962226520 (574h11m33.833309565s) [noise=true , address=77.90.31.10:4519     , piece_id=GGU2VNU5YF3I62VMF3YAIWZ2RYWLRZHXNU2RUV7JQT24X7TP2H3A, seg_pos={0 0}, upload=503383]: Uploading piece...
Dec 29 13:40:33.962542131 (574h11m33.833625166s) [noise=true , address=45.91.116.13:4153    , piece_id=LASX5P46Z7VSCUPVORZQ4CACRIT3KA33C6RZXCCSBRW5YY3O6U3A, seg_pos={0 0}, upload=503383]: Uploading piece...
Dec 29 13:40:33.962614599 (574h11m33.833697644s) [noise=true , address=145.79.227.52:4447   , piece_id=AWSSJDJAUOTWPZ6AKCIR6HPYZ2EJKJLSYGTP5ARCNOWOUTRBVTMQ, seg_pos={0 0}, upload=503383]: Uploading piece...
Dec 29 13:40:33.962801745 (574h11m33.83388479s) [noise=true , address=145.79.227.19:4077   , piece_id=VLCKHDO4HBTYEQVG4MKKF6MZI6HWHPAW2VYWDY3ZCQETFDJFS2HA, seg_pos={0 0}, upload=503383]: Uploading piece...
Dec 29 13:40:33.963072170 (574h11m33.834155215s) [noise=true , address=145.79.227.36:4517   , piece_id=LM36NFSPT4DTUYDLELRBHY5A7KBO7K5SCIJQSA4RWF7QVMFN7VSA, seg_pos={0 0}, upload=503383]: Uploading piece...
Dec 29 13:40:33.963482441 (574h11m33.834565476s) [noise=true , address=145.79.227.13:4804   , piece_id=TRV4TJHMQDRUGXNG2VTUJBKK3XWGGNHZBKJHM7LZQHTLAE66M7AQ, seg_pos={0 0}, upload=503383]: Uploading piece...
Dec 29 13:40:33.963859609 (574h11m33.834942655s) [noise=true , address=45.91.116.4:4937     , piece_id=R2N3A5QO3ML46CSCBAPFIRSMOOPL2OIRE5G7HNTZLYABLWIBPC5Q, seg_pos={0 0}, upload=503383]: Uploading piece...
Dec 29 13:40:33.964526028 (574h11m33.835609063s) [noise=true , address=45.91.116.12:4228    , piece_id=VJVD3CRBPE335HA7Y2SC4SPLPI2RAN3CYHFF2A2RJQHRLR7QDJVA, seg_pos={0 0}, upload=503383]: Uploading piece...
Dec 29 13:40:33.965601556 (574h11m33.836684591s) [noise=true , address=45.91.116.14:4189    , piece_id=GMOB6MV34TSIJPAR46GXQ74JMORKMHMKNR5QGI7OEKXZF7S2HPTQ, seg_pos={0 0}, upload=503383]: Uploading piece...
Dec 29 13:40:33.966391580 (574h11m33.837474615s) [noise=true , address=145.79.227.20:4437   , piece_id=5LP4ZZVD53EZZZ3SEGBYYN4VHKX2QFLSLDA7HUR7D6S3C2RXYSUA, seg_pos={0 0}, upload=503383]: Uploading piece...
Dec 29 13:40:33.966723192 (574h11m33.837806227s) [noise=true , address=45.91.116.5:4298     , piece_id=DL7LDVQNIWZP4UVFIS5SHQATIZQEY7VXVFPV5MUM2K5WXJOO3ZIQ, seg_pos={0 0}, upload=503383]: Uploading piece...
Dec 29 13:40:33.967039364 (574h11m33.838122409s) [noise=true , address=45.91.116.7:4010     , piece_id=C3XF2WVCC4P4OVOUXPCK6N53CXTM4UCEF5ZN6RDR5CPIYX3MJODQ, seg_pos={0 0}, upload=503383]: Uploading piece...
Dec 29 13:40:33.967498809 (574h11m33.838581854s) [noise=true , address=145.79.227.33:4869   , piece_id=7OAUIR4ED63YFSCHXHN4HSYSGBXQDR6GYXFRS5KZCYTBZWWWGUSA, seg_pos={0 0}, upload=503383]: Uploading piece...
Dec 29 13:40:35.656773507 (574h11m35.527856552s) [upload=503383]: Next returned nil segment
Dec 29 13:40:35.656789377 (574h11m35.527872412s) [upload=503383]: Waiting for error group managing segments...
Dec 29 13:40:35.944630668 (574h11m35.815713724s) [noise=true , address=145.79.227.2:4115    , piece_id=WVKUZECFJ24LFZTDJOMBFTLYDNRXD6M3YHRLKCPFIYL6MDE2FGWA, seg_pos={0 0}, upload=503383]: Done uploading piece. err: <nil>
Dec 29 13:40:35.945140009 (574h11m35.816223044s) [noise=true , address=145.79.227.13:4804   , piece_id=TRV4TJHMQDRUGXNG2VTUJBKK3XWGGNHZBKJHM7LZQHTLAE66M7AQ, seg_pos={0 0}, upload=503383]: Done uploading piece. err: <nil>
Dec 29 13:40:35.945732286 (574h11m35.816815322s) [noise=true , address=145.79.227.19:4077   , piece_id=VLCKHDO4HBTYEQVG4MKKF6MZI6HWHPAW2VYWDY3ZCQETFDJFS2HA, seg_pos={0 0}, upload=503383]: Done uploading piece. err: <nil>
Dec 29 13:40:35.946679831 (574h11m35.817762876s) [noise=true , address=145.79.227.36:4517   , piece_id=LM36NFSPT4DTUYDLELRBHY5A7KBO7K5SCIJQSA4RWF7QVMFN7VSA, seg_pos={0 0}, upload=503383]: Done uploading piece. err: <nil>
Dec 29 13:40:35.953101038 (574h11m35.824184073s) [noise=true , address=145.79.227.20:4437   , piece_id=5LP4ZZVD53EZZZ3SEGBYYN4VHKX2QFLSLDA7HUR7D6S3C2RXYSUA, seg_pos={0 0}, upload=503383]: Done uploading piece. err: <nil>
Dec 29 13:40:35.953387183 (574h11m35.824470228s) [noise=true , address=145.79.227.39:4014   , piece_id=L3XYIRTFGKKLGQ5DD33GIIBNUHPI4E63V2JWFMRQ2ZX2OQJ6LGZQ, seg_pos={0 0}, upload=503383]: Done uploading piece. err: <nil>
Dec 29 13:40:35.959157831 (574h11m35.830240866s) [noise=true , address=145.79.227.3:4149    , piece_id=S455YAPDXPDXBIQERWZSWKRD5XMZTKICA5JFTYNTGPT6BPTTM5PQ, seg_pos={0 0}, upload=503383]: Done uploading piece. err: <nil>
Dec 29 13:40:35.983014036 (574h11m35.854097071s) [noise=true , address=145.79.227.52:4447   , piece_id=AWSSJDJAUOTWPZ6AKCIR6HPYZ2EJKJLSYGTP5ARCNOWOUTRBVTMQ, seg_pos={0 0}, upload=503383]: Done uploading piece. err: <nil>
Dec 29 13:40:36.223613774 (574h11m36.094696809s) [noise=true , address=45.91.116.5:4298     , piece_id=DL7LDVQNIWZP4UVFIS5SHQATIZQEY7VXVFPV5MUM2K5WXJOO3ZIQ, seg_pos={0 0}, upload=503383]: Done uploading piece. err: <nil>
Dec 29 13:40:36.328551211 (574h11m36.199634246s) [noise=true , address=45.91.116.4:4937     , piece_id=R2N3A5QO3ML46CSCBAPFIRSMOOPL2OIRE5G7HNTZLYABLWIBPC5Q, seg_pos={0 0}, upload=503383]: Done uploading piece. err: <nil>
Dec 29 13:40:36.381207846 (574h11m36.252290881s) [noise=true , address=45.91.116.14:4189    , piece_id=GMOB6MV34TSIJPAR46GXQ74JMORKMHMKNR5QGI7OEKXZF7S2HPTQ, seg_pos={0 0}, upload=503383]: Done uploading piece. err: <nil>
Dec 29 13:40:36.575193915 (574h11m36.44627695s) [noise=true , address=45.91.116.12:4228    , piece_id=VJVD3CRBPE335HA7Y2SC4SPLPI2RAN3CYHFF2A2RJQHRLR7QDJVA, seg_pos={0 0}, upload=503383]: Done uploading piece. err: <nil>
Dec 29 13:40:36.597459952 (574h11m36.468542988s) [noise=true , address=77.90.31.10:4519     , piece_id=GGU2VNU5YF3I62VMF3YAIWZ2RYWLRZHXNU2RUV7JQT24X7TP2H3A, seg_pos={0 0}, upload=503383]: Done uploading piece. err: <nil>
Dec 29 13:40:36.597493977 (574h11m36.468577012s) [seg_pos={0 0}, upload=503383]: Segment reached optimal threshold of 13 pieces.
Dec 29 13:40:36.597701673 (574h11m36.468784708s) [noise=true , address=145.79.227.33:4869   , piece_id=7OAUIR4ED63YFSCHXHN4HSYSGBXQDR6GYXFRS5KZCYTBZWWWGUSA, seg_pos={0 0}, upload=503383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF
Dec 29 13:40:36.597708666 (574h11m36.468791701s) [noise=true , address=145.79.227.33:4869   , piece_id=7OAUIR4ED63YFSCHXHN4HSYSGBXQDR6GYXFRS5KZCYTBZWWWGUSA, seg_pos={0 0}, upload=503383]: Long tail cancellation due to optimal threshold
Dec 29 13:40:36.597923385 (574h11m36.46900643s) [noise=true , address=45.91.116.13:4153    , piece_id=LASX5P46Z7VSCUPVORZQ4CACRIT3KA33C6RZXCCSBRW5YY3O6U3A, seg_pos={0 0}, upload=503383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:43612->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:43612->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43612->45.91.116.13:4153: use of closed network connection
Dec 29 13:40:36.597930318 (574h11m36.469013363s) [noise=true , address=45.91.116.13:4153    , piece_id=LASX5P46Z7VSCUPVORZQ4CACRIT3KA33C6RZXCCSBRW5YY3O6U3A, seg_pos={0 0}, upload=503383]: Long tail cancellation due to optimal threshold
Dec 29 13:40:36.597933244 (574h11m36.469016279s) [noise=true , address=45.91.116.7:4010     , piece_id=C3XF2WVCC4P4OVOUXPCK6N53CXTM4UCEF5ZN6RDR5CPIYX3MJODQ, seg_pos={0 0}, upload=503383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:39890->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:39890->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39890->45.91.116.7:4010: use of closed network connection
Dec 29 13:40:36.597950877 (574h11m36.469033922s) [noise=true , address=45.91.116.7:4010     , piece_id=C3XF2WVCC4P4OVOUXPCK6N53CXTM4UCEF5ZN6RDR5CPIYX3MJODQ, seg_pos={0 0}, upload=503383]: Long tail cancellation due to optimal threshold
Dec 29 13:40:36.597987186 (574h11m36.469070231s) [seg_pos={0 0}, upload=503383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 13:40:36.598175304 (574h11m36.469258339s) [upload=503383]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 13:40:36.598181917 (574h11m36.469264953s) [upload=503383]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 13:40:36.886452687 (574h11m36.757535732s) [upload=503383]: Done uploading segments...
Dec 29 13:40:36.886479378 (574h11m36.757562423s) [upload=503383]: Upload finished. err: <nil>
Duration: 6.908403705s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [...X.....................XX.................................]
duration: 
56.833167998s
trace: 
https://jaeger.dev.storj.io/trace/77d26141ea11d228
FAIL (2025-12-29 13:41:27 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12J2weU7Bmsr3RdQQ9QMuadT96XReBkyxTeaLJ2SEqVGGFX1PWu, Piece ID: JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ) context deadline exceeded; piecestore: (Node ID: 125sm6iHV94mJFLWrKeyRMH4b1qSMo32nDuW8uQr71xUL7Cema2, Piece ID: INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA) context deadline exceeded; piecestore: (Node ID: 12i57FF97oEddYLZpNxSq5or9WLEquL8ZW1cVUuW4AThBbDqcCL, Piece ID: Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA) context deadline exceeded
======LOGS======
Q2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.692912334 (573h43m3.563995379s) [noise=true , address=77.90.31.16:4759     , piece_id=ARC4HRP57U5VQXZTT63QYLXBPZURI5VMJBIQZOFKGLJBQ3K3KGLQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.693322595 (573h43m3.56440564s) [noise=true , address=77.90.31.15:4711     , piece_id=JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.693739649 (573h43m3.564822694s) [noise=true , address=45.91.116.6:4981     , piece_id=INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.694315887 (573h43m3.565398942s) [noise=true , address=145.79.227.4:4220    , piece_id=SHHJ7EW7DTNOZXVBP6A64VIX7P6AQN62TBCUT3KTH4MIB6UXY3BA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.694765222 (573h43m3.565848267s) [noise=true , address=145.79.227.29:4952   , piece_id=DKYDB2R4DNCGH6T3SLRR2MOZZUPVZZVRL2BKGYOBYXJJVPSZEJ4Q, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.695202545 (573h43m3.56628559s) [noise=true , address=145.79.227.40:4058   , piece_id=KRHH2QZKSDPWTSSKYDVDWGKFJD3LPGP6U7KI2PPJ5T3TXYIUGJWA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.695687528 (573h43m3.566770584s) [noise=true , address=77.90.31.19:4857     , piece_id=Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.696135221 (573h43m3.567218276s) [noise=true , address=45.91.116.8:4051     , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.696703733 (573h43m3.567786778s) [noise=true , address=64.49.13.170:4720    , piece_id=HF6X3OT2EPPDE3DWAPWABVZUR3ERDBWXHG3NGNKEWZAD7G5AN4TA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.698418298 (573h43m3.569501343s) [noise=true , address=145.79.227.36:4520   , piece_id=MYJSXUH563JFUW5Q7YUIWL7466G6ASY5WXR3T2PGUFFCTRTOBYJA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.700198829 (573h43m3.571281874s) [noise=true , address=77.90.31.18:4821     , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.701993145 (573h43m3.57307619s) [noise=true , address=193.149.164.143:4699 , piece_id=UDFDCYO25MOJBTMDE6F2WILXE3HLOB75UJZ4LW7ZMNSCOBJI6UJA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.703884687 (573h43m3.574967732s) [noise=true , address=145.79.227.33:4893   , piece_id=XGLX3DFNAY4F4GPGCGZ4ONDJ7NK62SAKX3PBRWHWI74EF3ZFMUSA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.705687109 (573h43m3.576770154s) [noise=true , address=45.91.116.10:4094    , piece_id=6UUIZNTESBBWJMDQ5ZTCHU7S6YKM4GDKO4ZAC6T2FUTPVGFFDS3A, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.707557651 (573h43m3.578640696s) [noise=true , address=145.79.227.43:4164   , piece_id=J6Q4ZDOIIIBA2FXGZZTD6AYS7KGAWSRXXUBAPDPQWYKNU5GNXBBQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:05.168247719 (573h43m5.039330754s) [upload=502975]: Next returned nil segment
Dec 29 13:12:05.168259962 (573h43m5.039343007s) [upload=502975]: Waiting for error group managing segments...
Dec 29 13:12:05.558873127 (573h43m5.429956162s) [noise=true , address=145.79.227.4:4220    , piece_id=SHHJ7EW7DTNOZXVBP6A64VIX7P6AQN62TBCUT3KTH4MIB6UXY3BA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.563024312 (573h43m5.434107347s) [noise=true , address=64.49.13.170:4720    , piece_id=HF6X3OT2EPPDE3DWAPWABVZUR3ERDBWXHG3NGNKEWZAD7G5AN4TA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.564115719 (573h43m5.435198755s) [noise=true , address=145.79.227.36:4520   , piece_id=MYJSXUH563JFUW5Q7YUIWL7466G6ASY5WXR3T2PGUFFCTRTOBYJA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.593873402 (573h43m5.464956447s) [noise=true , address=145.79.227.29:4952   , piece_id=DKYDB2R4DNCGH6T3SLRR2MOZZUPVZZVRL2BKGYOBYXJJVPSZEJ4Q, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.898194604 (573h43m5.769277639s) [noise=true , address=45.91.116.10:4094    , piece_id=6UUIZNTESBBWJMDQ5ZTCHU7S6YKM4GDKO4ZAC6T2FUTPVGFFDS3A, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.912845292 (573h43m5.783928337s) [noise=true , address=77.90.31.19:4857     , piece_id=Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.426532207 (573h43m6.297615252s) [noise=true , address=193.149.164.143:4699 , piece_id=UDFDCYO25MOJBTMDE6F2WILXE3HLOB75UJZ4LW7ZMNSCOBJI6UJA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.449589562 (573h43m6.320672607s) [noise=true , address=77.90.31.15:4711     , piece_id=JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.453186841 (573h43m6.324269876s) [noise=true , address=145.79.227.40:4058   , piece_id=KRHH2QZKSDPWTSSKYDVDWGKFJD3LPGP6U7KI2PPJ5T3TXYIUGJWA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.510769276 (573h43m6.381852321s) [noise=true , address=145.79.227.43:4164   , piece_id=J6Q4ZDOIIIBA2FXGZZTD6AYS7KGAWSRXXUBAPDPQWYKNU5GNXBBQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.518868879 (573h43m6.389951924s) [noise=true , address=145.79.227.33:4893   , piece_id=XGLX3DFNAY4F4GPGCGZ4ONDJ7NK62SAKX3PBRWHWI74EF3ZFMUSA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.519238043 (573h43m6.390321098s) [noise=true , address=45.91.116.6:4981     , piece_id=INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.523211328 (573h43m6.394294373s) [noise=true , address=77.90.31.16:4759     , piece_id=ARC4HRP57U5VQXZTT63QYLXBPZURI5VMJBIQZOFKGLJBQ3K3KGLQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.767869783 (573h43m6.638952818s) [noise=true , address=145.79.227.34:4924   , piece_id=MAQXQTAGXQ7AEPUCESUSUGMHSDMMXS7HHKAEA7B3HW5X66PDUF5A, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.767889431 (573h43m6.638972476s) [seg_pos={0 0}, upload=502975]: Segment reached optimal threshold of 14 pieces.
Dec 29 13:12:06.768102116 (573h43m6.639185161s) [noise=true , address=45.91.116.8:4051     , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1preT33ei4yVXXYVLHhaUbHbiyL14oHFEiof6cCWPig83W6CT3): protocol: expected piece hash; context canceled; EOF
Dec 29 13:12:06.768108257 (573h43m6.639191302s) [noise=true , address=45.91.116.8:4051     , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768219208 (573h43m6.639302254s) [noise=true , address=77.90.31.18:4821     , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH): protocol: expected piece hash; context canceled; EOF
Dec 29 13:12:06.768224789 (573h43m6.639307824s) [noise=true , address=77.90.31.18:4821     , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768384213 (573h43m6.639467258s) [noise=true , address=64.49.13.169:4711    , piece_id=4F4T4YKXMT7NF2C5KBUGJHQJGZL4VCKNKYB266PCVI3UACHJXZ5Q, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection
Dec 29 13:12:06.768399261 (573h43m6.639482296s) [noise=true , address=64.49.13.169:4711    , piece_id=4F4T4YKXMT7NF2C5KBUGJHQJGZL4VCKNKYB266PCVI3UACHJXZ5Q, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768732636 (573h43m6.639815671s) [noise=true , address=64.49.13.169:4709    , piece_id=TBHHX5ES2A6NQRGP25HIC7J7IBWLOJGQ2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:35310->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:35310->64.49.13.169:4709: use of closed network connection
Dec 29 13:12:06.768745460 (573h43m6.639828505s) [noise=true , address=64.49.13.169:4709    , piece_id=TBHHX5ES2A6NQRGP25HIC7J7IBWLOJGQ2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768784586 (573h43m6.639867631s) [seg_pos={0 0}, upload=502975]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 13:12:06.768880217 (573h43m6.639963252s) [upload=502975]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 13:12:06.768884906 (573h43m6.639967941s) [upload=502975]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 13:12:07.067188330 (573h43m6.938271365s) [upload=502975]: Done uploading segments...
Dec 29 13:12:07.067251099 (573h43m6.938334134s) [upload=502975]: Upload finished. err: <nil>
Duration: 56.83318444s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
6 out of 60 is failed (threshold=3) [.....X....X.X....X.X....................................X...]
duration: 
23.197364083s
trace: 
https://jaeger.dev.storj.io/trace/1f23099550b2a511
FAIL (2025-12-29 13:41:07 UTC) 
orbiter warning: context deadline exceeded
Duration: 23.19737867s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
32 out of 60 is failed (threshold=3) [X.XXX.X.X.X.XX..X.XXX.XX..X.XX..XXX...X..X...X...X.XXXX.X.XX]
duration: 
30.005862522s
trace: 
https://jaeger.dev.storj.io/trace/11c1b2787abeda0a
FAIL (2025-12-29 13:41:03 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
segments...
Dec 29 13:40:33.713978579 (574h11m33.585061624s) [upload=503383]: Got next segment. Inline: false
Dec 29 13:40:33.714036390 (574h11m33.585119435s) [upload=503383]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 13:40:33.714041389 (574h11m33.585124424s) [upload=503383]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 13:40:33.960974115 (574h11m33.83205715s) [seg_pos={0 0}, upload=503383]: Begin upload segment...
Dec 29 13:40:33.961059237 (574h11m33.832142302s) [noise=true , address=145.79.227.2:4115    , piece_id=WVKUZECFJ24LFZTDJOMBFTLYDNRXD6M3YHRLKCPFIYL6MDE2FGWA, seg_pos={0 0}, upload=503383]: Uploading piece...
Dec 29 13:40:33.961114292 (574h11m33.832197337s) [seg_pos={0 0}, upload=503383]: Done begin upload segment.
Dec 29 13:40:33.961256663 (574h11m33.832339708s) [noise=true , address=145.79.227.39:4014   , piece_id=L3XYIRTFGKKLGQ5DD33GIIBNUHPI4E63V2JWFMRQ2ZX2OQJ6LGZQ, seg_pos={0 0}, upload=503383]: Uploading piece...
Dec 29 13:40:33.961769740 (574h11m33.832852775s) [noise=true , address=145.79.227.3:4149    , piece_id=S455YAPDXPDXBIQERWZSWKRD5XMZTKICA5JFTYNTGPT6BPTTM5PQ, seg_pos={0 0}, upload=503383]: Uploading piece...
Dec 29 13:40:33.962226520 (574h11m33.833309565s) [noise=true , address=77.90.31.10:4519     , piece_id=GGU2VNU5YF3I62VMF3YAIWZ2RYWLRZHXNU2RUV7JQT24X7TP2H3A, seg_pos={0 0}, upload=503383]: Uploading piece...
Dec 29 13:40:33.962542131 (574h11m33.833625166s) [noise=true , address=45.91.116.13:4153    , piece_id=LASX5P46Z7VSCUPVORZQ4CACRIT3KA33C6RZXCCSBRW5YY3O6U3A, seg_pos={0 0}, upload=503383]: Uploading piece...
Dec 29 13:40:33.962614599 (574h11m33.833697644s) [noise=true , address=145.79.227.52:4447   , piece_id=AWSSJDJAUOTWPZ6AKCIR6HPYZ2EJKJLSYGTP5ARCNOWOUTRBVTMQ, seg_pos={0 0}, upload=503383]: Uploading piece...
Dec 29 13:40:33.962801745 (574h11m33.83388479s) [noise=true , address=145.79.227.19:4077   , piece_id=VLCKHDO4HBTYEQVG4MKKF6MZI6HWHPAW2VYWDY3ZCQETFDJFS2HA, seg_pos={0 0}, upload=503383]: Uploading piece...
Dec 29 13:40:33.963072170 (574h11m33.834155215s) [noise=true , address=145.79.227.36:4517   , piece_id=LM36NFSPT4DTUYDLELRBHY5A7KBO7K5SCIJQSA4RWF7QVMFN7VSA, seg_pos={0 0}, upload=503383]: Uploading piece...
Dec 29 13:40:33.963482441 (574h11m33.834565476s) [noise=true , address=145.79.227.13:4804   , piece_id=TRV4TJHMQDRUGXNG2VTUJBKK3XWGGNHZBKJHM7LZQHTLAE66M7AQ, seg_pos={0 0}, upload=503383]: Uploading piece...
Dec 29 13:40:33.963859609 (574h11m33.834942655s) [noise=true , address=45.91.116.4:4937     , piece_id=R2N3A5QO3ML46CSCBAPFIRSMOOPL2OIRE5G7HNTZLYABLWIBPC5Q, seg_pos={0 0}, upload=503383]: Uploading piece...
Dec 29 13:40:33.964526028 (574h11m33.835609063s) [noise=true , address=45.91.116.12:4228    , piece_id=VJVD3CRBPE335HA7Y2SC4SPLPI2RAN3CYHFF2A2RJQHRLR7QDJVA, seg_pos={0 0}, upload=503383]: Uploading piece...
Dec 29 13:40:33.965601556 (574h11m33.836684591s) [noise=true , address=45.91.116.14:4189    , piece_id=GMOB6MV34TSIJPAR46GXQ74JMORKMHMKNR5QGI7OEKXZF7S2HPTQ, seg_pos={0 0}, upload=503383]: Uploading piece...
Dec 29 13:40:33.966391580 (574h11m33.837474615s) [noise=true , address=145.79.227.20:4437   , piece_id=5LP4ZZVD53EZZZ3SEGBYYN4VHKX2QFLSLDA7HUR7D6S3C2RXYSUA, seg_pos={0 0}, upload=503383]: Uploading piece...
Dec 29 13:40:33.966723192 (574h11m33.837806227s) [noise=true , address=45.91.116.5:4298     , piece_id=DL7LDVQNIWZP4UVFIS5SHQATIZQEY7VXVFPV5MUM2K5WXJOO3ZIQ, seg_pos={0 0}, upload=503383]: Uploading piece...
Dec 29 13:40:33.967039364 (574h11m33.838122409s) [noise=true , address=45.91.116.7:4010     , piece_id=C3XF2WVCC4P4OVOUXPCK6N53CXTM4UCEF5ZN6RDR5CPIYX3MJODQ, seg_pos={0 0}, upload=503383]: Uploading piece...
Dec 29 13:40:33.967498809 (574h11m33.838581854s) [noise=true , address=145.79.227.33:4869   , piece_id=7OAUIR4ED63YFSCHXHN4HSYSGBXQDR6GYXFRS5KZCYTBZWWWGUSA, seg_pos={0 0}, upload=503383]: Uploading piece...
Dec 29 13:40:35.656773507 (574h11m35.527856552s) [upload=503383]: Next returned nil segment
Dec 29 13:40:35.656789377 (574h11m35.527872412s) [upload=503383]: Waiting for error group managing segments...
Dec 29 13:40:35.944630668 (574h11m35.815713724s) [noise=true , address=145.79.227.2:4115    , piece_id=WVKUZECFJ24LFZTDJOMBFTLYDNRXD6M3YHRLKCPFIYL6MDE2FGWA, seg_pos={0 0}, upload=503383]: Done uploading piece. err: <nil>
Dec 29 13:40:35.945140009 (574h11m35.816223044s) [noise=true , address=145.79.227.13:4804   , piece_id=TRV4TJHMQDRUGXNG2VTUJBKK3XWGGNHZBKJHM7LZQHTLAE66M7AQ, seg_pos={0 0}, upload=503383]: Done uploading piece. err: <nil>
Dec 29 13:40:35.945732286 (574h11m35.816815322s) [noise=true , address=145.79.227.19:4077   , piece_id=VLCKHDO4HBTYEQVG4MKKF6MZI6HWHPAW2VYWDY3ZCQETFDJFS2HA, seg_pos={0 0}, upload=503383]: Done uploading piece. err: <nil>
Dec 29 13:40:35.946679831 (574h11m35.817762876s) [noise=true , address=145.79.227.36:4517   , piece_id=LM36NFSPT4DTUYDLELRBHY5A7KBO7K5SCIJQSA4RWF7QVMFN7VSA, seg_pos={0 0}, upload=503383]: Done uploading piece. err: <nil>
Dec 29 13:40:35.953101038 (574h11m35.824184073s) [noise=true , address=145.79.227.20:4437   , piece_id=5LP4ZZVD53EZZZ3SEGBYYN4VHKX2QFLSLDA7HUR7D6S3C2RXYSUA, seg_pos={0 0}, upload=503383]: Done uploading piece. err: <nil>
Dec 29 13:40:35.953387183 (574h11m35.824470228s) [noise=true , address=145.79.227.39:4014   , piece_id=L3XYIRTFGKKLGQ5DD33GIIBNUHPI4E63V2JWFMRQ2ZX2OQJ6LGZQ, seg_pos={0 0}, upload=503383]: Done uploading piece. err: <nil>
Dec 29 13:40:35.959157831 (574h11m35.830240866s) [noise=true , address=145.79.227.3:4149    , piece_id=S455YAPDXPDXBIQERWZSWKRD5XMZTKICA5JFTYNTGPT6BPTTM5PQ, seg_pos={0 0}, upload=503383]: Done uploading piece. err: <nil>
Dec 29 13:40:35.983014036 (574h11m35.854097071s) [noise=true , address=145.79.227.52:4447   , piece_id=AWSSJDJAUOTWPZ6AKCIR6HPYZ2EJKJLSYGTP5ARCNOWOUTRBVTMQ, seg_pos={0 0}, upload=503383]: Done uploading piece. err: <nil>
Dec 29 13:40:36.223613774 (574h11m36.094696809s) [noise=true , address=45.91.116.5:4298     , piece_id=DL7LDVQNIWZP4UVFIS5SHQATIZQEY7VXVFPV5MUM2K5WXJOO3ZIQ, seg_pos={0 0}, upload=503383]: Done uploading piece. err: <nil>
Dec 29 13:40:36.328551211 (574h11m36.199634246s) [noise=true , address=45.91.116.4:4937     , piece_id=R2N3A5QO3ML46CSCBAPFIRSMOOPL2OIRE5G7HNTZLYABLWIBPC5Q, seg_pos={0 0}, upload=503383]: Done uploading piece. err: <nil>
Dec 29 13:40:36.381207846 (574h11m36.252290881s) [noise=true , address=45.91.116.14:4189    , piece_id=GMOB6MV34TSIJPAR46GXQ74JMORKMHMKNR5QGI7OEKXZF7S2HPTQ, seg_pos={0 0}, upload=503383]: Done uploading piece. err: <nil>
Dec 29 13:40:36.575193915 (574h11m36.44627695s) [noise=true , address=45.91.116.12:4228    , piece_id=VJVD3CRBPE335HA7Y2SC4SPLPI2RAN3CYHFF2A2RJQHRLR7QDJVA, seg_pos={0 0}, upload=503383]: Done uploading piece. err: <nil>
Dec 29 13:40:36.597459952 (574h11m36.468542988s) [noise=true , address=77.90.31.10:4519     , piece_id=GGU2VNU5YF3I62VMF3YAIWZ2RYWLRZHXNU2RUV7JQT24X7TP2H3A, seg_pos={0 0}, upload=503383]: Done uploading piece. err: <nil>
Dec 29 13:40:36.597493977 (574h11m36.468577012s) [seg_pos={0 0}, upload=503383]: Segment reached optimal threshold of 13 pieces.
Dec 29 13:40:36.597701673 (574h11m36.468784708s) [noise=true , address=145.79.227.33:4869   , piece_id=7OAUIR4ED63YFSCHXHN4HSYSGBXQDR6GYXFRS5KZCYTBZWWWGUSA, seg_pos={0 0}, upload=503383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF
Dec 29 13:40:36.597708666 (574h11m36.468791701s) [noise=true , address=145.79.227.33:4869   , piece_id=7OAUIR4ED63YFSCHXHN4HSYSGBXQDR6GYXFRS5KZCYTBZWWWGUSA, seg_pos={0 0}, upload=503383]: Long tail cancellation due to optimal threshold
Dec 29 13:40:36.597923385 (574h11m36.46900643s) [noise=true , address=45.91.116.13:4153    , piece_id=LASX5P46Z7VSCUPVORZQ4CACRIT3KA33C6RZXCCSBRW5YY3O6U3A, seg_pos={0 0}, upload=503383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:43612->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:43612->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43612->45.91.116.13:4153: use of closed network connection
Dec 29 13:40:36.597930318 (574h11m36.469013363s) [noise=true , address=45.91.116.13:4153    , piece_id=LASX5P46Z7VSCUPVORZQ4CACRIT3KA33C6RZXCCSBRW5YY3O6U3A, seg_pos={0 0}, upload=503383]: Long tail cancellation due to optimal threshold
Dec 29 13:40:36.597933244 (574h11m36.469016279s) [noise=true , address=45.91.116.7:4010     , piece_id=C3XF2WVCC4P4OVOUXPCK6N53CXTM4UCEF5ZN6RDR5CPIYX3MJODQ, seg_pos={0 0}, upload=503383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:39890->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:39890->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39890->45.91.116.7:4010: use of closed network connection
Dec 29 13:40:36.597950877 (574h11m36.469033922s) [noise=true , address=45.91.116.7:4010     , piece_id=C3XF2WVCC4P4OVOUXPCK6N53CXTM4UCEF5ZN6RDR5CPIYX3MJODQ, seg_pos={0 0}, upload=503383]: Long tail cancellation due to optimal threshold
Dec 29 13:40:36.597987186 (574h11m36.469070231s) [seg_pos={0 0}, upload=503383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 13:40:36.598175304 (574h11m36.469258339s) [upload=503383]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 13:40:36.598181917 (574h11m36.469264953s) [upload=503383]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 13:40:36.886452687 (574h11m36.757535732s) [upload=503383]: Done uploading segments...
Dec 29 13:40:36.886479378 (574h11m36.757562423s) [upload=503383]: Upload finished. err: <nil>
Duration: 30.005884492s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)
antiblip: 
1 out of 60 is failed (threshold=3) [..........................................................X.]
duration: 
19.294606973s
trace: 
https://jaeger.dev.storj.io/trace/1df9ff77465ee3c2
FAIL (2025-12-29 13:41:02 UTC) 
orbiter warning: context deadline exceeded
Duration: 19.294626349s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
54 out of 60 is failed (threshold=6) [XX.XX.XXXXXXX.XX.XXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXX.XXXXXXX]
duration: 
5.001293469s
trace: 
https://jaeger.dev.storj.io/trace/4d0eb9d499c456bd
FAIL (2025-12-29 13:40:58 UTC) 
orbiter warning: context deadline exceeded
Duration: 5.001314971s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)
antiblip: 
1 out of 60 is failed (threshold=3) [...........................................................X]
duration: 
1m0.000835242s
trace: 
https://jaeger.dev.storj.io/trace/1df9ff77465ee3c2
FAIL (2025-12-29 13:40:42 UTC) 
orbiter warning: context deadline exceeded
Duration: 1m0.000862382s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [....X.....................XX................................]
duration: 
52.940013685s
trace: 
https://jaeger.dev.storj.io/trace/77d26141ea11d228
FAIL (2025-12-29 13:40:14 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12J2weU7Bmsr3RdQQ9QMuadT96XReBkyxTeaLJ2SEqVGGFX1PWu, Piece ID: JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ) context deadline exceeded; piecestore: (Node ID: 125sm6iHV94mJFLWrKeyRMH4b1qSMo32nDuW8uQr71xUL7Cema2, Piece ID: INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA) context deadline exceeded; piecestore: (Node ID: 12i57FF97oEddYLZpNxSq5or9WLEquL8ZW1cVUuW4AThBbDqcCL, Piece ID: Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA) context deadline exceeded
======LOGS======
Q2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.692912334 (573h43m3.563995379s) [noise=true , address=77.90.31.16:4759     , piece_id=ARC4HRP57U5VQXZTT63QYLXBPZURI5VMJBIQZOFKGLJBQ3K3KGLQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.693322595 (573h43m3.56440564s) [noise=true , address=77.90.31.15:4711     , piece_id=JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.693739649 (573h43m3.564822694s) [noise=true , address=45.91.116.6:4981     , piece_id=INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.694315887 (573h43m3.565398942s) [noise=true , address=145.79.227.4:4220    , piece_id=SHHJ7EW7DTNOZXVBP6A64VIX7P6AQN62TBCUT3KTH4MIB6UXY3BA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.694765222 (573h43m3.565848267s) [noise=true , address=145.79.227.29:4952   , piece_id=DKYDB2R4DNCGH6T3SLRR2MOZZUPVZZVRL2BKGYOBYXJJVPSZEJ4Q, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.695202545 (573h43m3.56628559s) [noise=true , address=145.79.227.40:4058   , piece_id=KRHH2QZKSDPWTSSKYDVDWGKFJD3LPGP6U7KI2PPJ5T3TXYIUGJWA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.695687528 (573h43m3.566770584s) [noise=true , address=77.90.31.19:4857     , piece_id=Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.696135221 (573h43m3.567218276s) [noise=true , address=45.91.116.8:4051     , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.696703733 (573h43m3.567786778s) [noise=true , address=64.49.13.170:4720    , piece_id=HF6X3OT2EPPDE3DWAPWABVZUR3ERDBWXHG3NGNKEWZAD7G5AN4TA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.698418298 (573h43m3.569501343s) [noise=true , address=145.79.227.36:4520   , piece_id=MYJSXUH563JFUW5Q7YUIWL7466G6ASY5WXR3T2PGUFFCTRTOBYJA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.700198829 (573h43m3.571281874s) [noise=true , address=77.90.31.18:4821     , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.701993145 (573h43m3.57307619s) [noise=true , address=193.149.164.143:4699 , piece_id=UDFDCYO25MOJBTMDE6F2WILXE3HLOB75UJZ4LW7ZMNSCOBJI6UJA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.703884687 (573h43m3.574967732s) [noise=true , address=145.79.227.33:4893   , piece_id=XGLX3DFNAY4F4GPGCGZ4ONDJ7NK62SAKX3PBRWHWI74EF3ZFMUSA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.705687109 (573h43m3.576770154s) [noise=true , address=45.91.116.10:4094    , piece_id=6UUIZNTESBBWJMDQ5ZTCHU7S6YKM4GDKO4ZAC6T2FUTPVGFFDS3A, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.707557651 (573h43m3.578640696s) [noise=true , address=145.79.227.43:4164   , piece_id=J6Q4ZDOIIIBA2FXGZZTD6AYS7KGAWSRXXUBAPDPQWYKNU5GNXBBQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:05.168247719 (573h43m5.039330754s) [upload=502975]: Next returned nil segment
Dec 29 13:12:05.168259962 (573h43m5.039343007s) [upload=502975]: Waiting for error group managing segments...
Dec 29 13:12:05.558873127 (573h43m5.429956162s) [noise=true , address=145.79.227.4:4220    , piece_id=SHHJ7EW7DTNOZXVBP6A64VIX7P6AQN62TBCUT3KTH4MIB6UXY3BA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.563024312 (573h43m5.434107347s) [noise=true , address=64.49.13.170:4720    , piece_id=HF6X3OT2EPPDE3DWAPWABVZUR3ERDBWXHG3NGNKEWZAD7G5AN4TA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.564115719 (573h43m5.435198755s) [noise=true , address=145.79.227.36:4520   , piece_id=MYJSXUH563JFUW5Q7YUIWL7466G6ASY5WXR3T2PGUFFCTRTOBYJA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.593873402 (573h43m5.464956447s) [noise=true , address=145.79.227.29:4952   , piece_id=DKYDB2R4DNCGH6T3SLRR2MOZZUPVZZVRL2BKGYOBYXJJVPSZEJ4Q, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.898194604 (573h43m5.769277639s) [noise=true , address=45.91.116.10:4094    , piece_id=6UUIZNTESBBWJMDQ5ZTCHU7S6YKM4GDKO4ZAC6T2FUTPVGFFDS3A, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.912845292 (573h43m5.783928337s) [noise=true , address=77.90.31.19:4857     , piece_id=Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.426532207 (573h43m6.297615252s) [noise=true , address=193.149.164.143:4699 , piece_id=UDFDCYO25MOJBTMDE6F2WILXE3HLOB75UJZ4LW7ZMNSCOBJI6UJA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.449589562 (573h43m6.320672607s) [noise=true , address=77.90.31.15:4711     , piece_id=JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.453186841 (573h43m6.324269876s) [noise=true , address=145.79.227.40:4058   , piece_id=KRHH2QZKSDPWTSSKYDVDWGKFJD3LPGP6U7KI2PPJ5T3TXYIUGJWA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.510769276 (573h43m6.381852321s) [noise=true , address=145.79.227.43:4164   , piece_id=J6Q4ZDOIIIBA2FXGZZTD6AYS7KGAWSRXXUBAPDPQWYKNU5GNXBBQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.518868879 (573h43m6.389951924s) [noise=true , address=145.79.227.33:4893   , piece_id=XGLX3DFNAY4F4GPGCGZ4ONDJ7NK62SAKX3PBRWHWI74EF3ZFMUSA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.519238043 (573h43m6.390321098s) [noise=true , address=45.91.116.6:4981     , piece_id=INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.523211328 (573h43m6.394294373s) [noise=true , address=77.90.31.16:4759     , piece_id=ARC4HRP57U5VQXZTT63QYLXBPZURI5VMJBIQZOFKGLJBQ3K3KGLQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.767869783 (573h43m6.638952818s) [noise=true , address=145.79.227.34:4924   , piece_id=MAQXQTAGXQ7AEPUCESUSUGMHSDMMXS7HHKAEA7B3HW5X66PDUF5A, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.767889431 (573h43m6.638972476s) [seg_pos={0 0}, upload=502975]: Segment reached optimal threshold of 14 pieces.
Dec 29 13:12:06.768102116 (573h43m6.639185161s) [noise=true , address=45.91.116.8:4051     , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1preT33ei4yVXXYVLHhaUbHbiyL14oHFEiof6cCWPig83W6CT3): protocol: expected piece hash; context canceled; EOF
Dec 29 13:12:06.768108257 (573h43m6.639191302s) [noise=true , address=45.91.116.8:4051     , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768219208 (573h43m6.639302254s) [noise=true , address=77.90.31.18:4821     , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH): protocol: expected piece hash; context canceled; EOF
Dec 29 13:12:06.768224789 (573h43m6.639307824s) [noise=true , address=77.90.31.18:4821     , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768384213 (573h43m6.639467258s) [noise=true , address=64.49.13.169:4711    , piece_id=4F4T4YKXMT7NF2C5KBUGJHQJGZL4VCKNKYB266PCVI3UACHJXZ5Q, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection
Dec 29 13:12:06.768399261 (573h43m6.639482296s) [noise=true , address=64.49.13.169:4711    , piece_id=4F4T4YKXMT7NF2C5KBUGJHQJGZL4VCKNKYB266PCVI3UACHJXZ5Q, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768732636 (573h43m6.639815671s) [noise=true , address=64.49.13.169:4709    , piece_id=TBHHX5ES2A6NQRGP25HIC7J7IBWLOJGQ2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:35310->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:35310->64.49.13.169:4709: use of closed network connection
Dec 29 13:12:06.768745460 (573h43m6.639828505s) [noise=true , address=64.49.13.169:4709    , piece_id=TBHHX5ES2A6NQRGP25HIC7J7IBWLOJGQ2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768784586 (573h43m6.639867631s) [seg_pos={0 0}, upload=502975]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 13:12:06.768880217 (573h43m6.639963252s) [upload=502975]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 13:12:06.768884906 (573h43m6.639967941s) [upload=502975]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 13:12:07.067188330 (573h43m6.938271365s) [upload=502975]: Done uploading segments...
Dec 29 13:12:07.067251099 (573h43m6.938334134s) [upload=502975]: Upload finished. err: <nil>
Duration: 52.940049443s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
6 out of 60 is failed (threshold=3) [......X....X.X....X.X....................................X..]
duration: 
22.061153396s
trace: 
https://jaeger.dev.storj.io/trace/1f23099550b2a511
FAIL (2025-12-29 13:40:09 UTC) 
orbiter warning: context deadline exceeded
Duration: 22.06118121s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
31 out of 60 is failed (threshold=3) [.X.XXX.X.X.X.XX..X.XXX.XX..X.XX..XXX...X..X...X...X.XXXX.X.X]
duration: 
30.003383291s
trace: 
https://jaeger.dev.storj.io/trace/430e4bbbbd83734a
FAIL (2025-12-29 13:39:56 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
gments...
Dec 29 13:39:27.395732113 (574h10m27.266815148s) [upload=503371]: Got next segment. Inline: false
Dec 29 13:39:27.395777118 (574h10m27.266860153s) [upload=503371]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 13:39:27.395781748 (574h10m27.266864783s) [upload=503371]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 13:39:27.642742127 (574h10m27.513825182s) [seg_pos={0 0}, upload=503371]: Begin upload segment...
Dec 29 13:39:27.642791490 (574h10m27.513874535s) [seg_pos={0 0}, upload=503371]: Done begin upload segment.
Dec 29 13:39:27.642825154 (574h10m27.513908189s) [noise=true , address=77.90.31.15:4697     , piece_id=UYPKV3GDMG2VY4UJKHQW4PLRUUVIOUSD4RRAIDINSA24BDT6ZMLQ, seg_pos={0 0}, upload=503371]: Uploading piece...
Dec 29 13:39:27.642833671 (574h10m27.513916716s) [noise=true , address=45.91.116.11:4117    , piece_id=XEA6MMRD5C7FQXZMBYRBS3PKUVCLGH66JBRN57YUXLQVCYXM6YXA, seg_pos={0 0}, upload=503371]: Uploading piece...
Dec 29 13:39:27.642908844 (574h10m27.513991889s) [noise=true , address=45.91.116.8:4045     , piece_id=ADUH2TEBPHYJGWPS7ANWHTSYZPPPDAY2HETVIQXVF4W2FXU742CA, seg_pos={0 0}, upload=503371]: Uploading piece...
Dec 29 13:39:27.643181302 (574h10m27.514264357s) [noise=true , address=145.79.227.11:4729   , piece_id=I3H3D3M5K3NUWIIOARZK3OD4E4KXREJRH5DBQBTE5UHG3XTN5KNA, seg_pos={0 0}, upload=503371]: Uploading piece...
Dec 29 13:39:27.643714969 (574h10m27.514798004s) [noise=true , address=145.79.227.42:4123   , piece_id=YBP7ZDRSYHJQIRFCBXLNMNZMVCM763MHBNYCHWNAQTZGUAKC3EYA, seg_pos={0 0}, upload=503371]: Uploading piece...
Dec 29 13:39:27.644185044 (574h10m27.515268089s) [noise=true , address=145.79.227.6:4258    , piece_id=K5KDQ4GYWF7T6N23N37KEAV63FUOUDDIFTLO7ZCLGLCT6GAWKCJQ, seg_pos={0 0}, upload=503371]: Uploading piece...
Dec 29 13:39:27.644550039 (574h10m27.515633084s) [noise=true , address=77.90.31.19:4841     , piece_id=NNGE6HDMWUG2FBAPKCNSZJWWJTMSHRYMETXB45GCF5ENV5NPBIAQ, seg_pos={0 0}, upload=503371]: Uploading piece...
Dec 29 13:39:27.644946584 (574h10m27.516029629s) [noise=true , address=145.79.227.3:4150    , piece_id=C3FTP4FYRLAJZAJ6DREANLHYVWJAWVYILIP5QSDSDMVVBSG4CJ3A, seg_pos={0 0}, upload=503371]: Uploading piece...
Dec 29 13:39:27.645392353 (574h10m27.516475398s) [noise=true , address=145.79.227.44:4193   , piece_id=CLIAY5DZK3EP6Q7MC5OWGNF27FWPXFDHCA2VKFVUWT6L53RQG4IA, seg_pos={0 0}, upload=503371]: Uploading piece...
Dec 29 13:39:27.645896382 (574h10m27.516979427s) [noise=true , address=77.90.31.17:4773     , piece_id=7QLG4MWCALBSVPCMSV75DRR2H3DIEWLISYWASSZO7IXPLBPQQG5A, seg_pos={0 0}, upload=503371]: Uploading piece...
Dec 29 13:39:27.645905529 (574h10m27.516988564s) [noise=true , address=145.79.227.14:4837   , piece_id=7FJUJC7N6ZFZZVMZ4XWPMGF5XLLW5MZSU55EABYRNH5RUBOKTSLA, seg_pos={0 0}, upload=503371]: Uploading piece...
Dec 29 13:39:27.646590985 (574h10m27.51767403s) [noise=true , address=145.79.227.41:4086   , piece_id=YKSFUUHMDRTTAPJF5VIOJPIWXBQLVGW26L5MAVFZLLVDBHM5QJQQ, seg_pos={0 0}, upload=503371]: Uploading piece...
Dec 29 13:39:27.647266943 (574h10m27.518349988s) [noise=true , address=45.91.116.13:4154    , piece_id=VPGRYHVJGCOVRB2RI4P3T7V5WLFLQ452IABYGJR2RC2LXTKDXUJA, seg_pos={0 0}, upload=503371]: Uploading piece...
Dec 29 13:39:27.647960031 (574h10m27.519043077s) [noise=true , address=45.91.116.9:4263     , piece_id=BLFARF6WFKLUW3Y4OZBTYQMNECLFBVLWNOHANH3YQ7F3G3OVL3FA, seg_pos={0 0}, upload=503371]: Uploading piece...
Dec 29 13:39:27.648551237 (574h10m27.519634282s) [noise=true , address=145.79.227.2:4113    , piece_id=Q2B2QIDFP3MGXBKSHKQR527TSGHJTRZMKXPKN55QJBTJ3V6MKD5A, seg_pos={0 0}, upload=503371]: Uploading piece...
Dec 29 13:39:27.649168803 (574h10m27.520251848s) [noise=true , address=145.79.227.48:4302   , piece_id=YSQX3DATHEZVRWZXCCCK245LOYO6SDYUJLXQGMA4ZJCBYPHREKFQ, seg_pos={0 0}, upload=503371]: Uploading piece...
Dec 29 13:39:29.354095964 (574h10m29.225179009s) [upload=503371]: Next returned nil segment
Dec 29 13:39:29.354234468 (574h10m29.225317513s) [upload=503371]: Waiting for error group managing segments...
Dec 29 13:39:29.636111052 (574h10m29.507194087s) [noise=true , address=145.79.227.11:4729   , piece_id=I3H3D3M5K3NUWIIOARZK3OD4E4KXREJRH5DBQBTE5UHG3XTN5KNA, seg_pos={0 0}, upload=503371]: Done uploading piece. err: <nil>
Dec 29 13:39:29.638650337 (574h10m29.509733383s) [noise=true , address=145.79.227.3:4150    , piece_id=C3FTP4FYRLAJZAJ6DREANLHYVWJAWVYILIP5QSDSDMVVBSG4CJ3A, seg_pos={0 0}, upload=503371]: Done uploading piece. err: <nil>
Dec 29 13:39:29.639289235 (574h10m29.51037227s) [noise=true , address=145.79.227.2:4113    , piece_id=Q2B2QIDFP3MGXBKSHKQR527TSGHJTRZMKXPKN55QJBTJ3V6MKD5A, seg_pos={0 0}, upload=503371]: Done uploading piece. err: <nil>
Dec 29 13:39:29.644049128 (574h10m29.515132163s) [noise=true , address=145.79.227.42:4123   , piece_id=YBP7ZDRSYHJQIRFCBXLNMNZMVCM763MHBNYCHWNAQTZGUAKC3EYA, seg_pos={0 0}, upload=503371]: Done uploading piece. err: <nil>
Dec 29 13:39:29.644141253 (574h10m29.515224298s) [noise=true , address=145.79.227.48:4302   , piece_id=YSQX3DATHEZVRWZXCCCK245LOYO6SDYUJLXQGMA4ZJCBYPHREKFQ, seg_pos={0 0}, upload=503371]: Done uploading piece. err: <nil>
Dec 29 13:39:29.652461396 (574h10m29.523544431s) [noise=true , address=145.79.227.6:4258    , piece_id=K5KDQ4GYWF7T6N23N37KEAV63FUOUDDIFTLO7ZCLGLCT6GAWKCJQ, seg_pos={0 0}, upload=503371]: Done uploading piece. err: <nil>
Dec 29 13:39:29.662069382 (574h10m29.533152427s) [noise=true , address=145.79.227.41:4086   , piece_id=YKSFUUHMDRTTAPJF5VIOJPIWXBQLVGW26L5MAVFZLLVDBHM5QJQQ, seg_pos={0 0}, upload=503371]: Done uploading piece. err: <nil>
Dec 29 13:39:29.663633108 (574h10m29.534716143s) [noise=true , address=145.79.227.14:4837   , piece_id=7FJUJC7N6ZFZZVMZ4XWPMGF5XLLW5MZSU55EABYRNH5RUBOKTSLA, seg_pos={0 0}, upload=503371]: Done uploading piece. err: <nil>
Dec 29 13:39:29.842976225 (574h10m29.71405926s) [noise=true , address=77.90.31.15:4697     , piece_id=UYPKV3GDMG2VY4UJKHQW4PLRUUVIOUSD4RRAIDINSA24BDT6ZMLQ, seg_pos={0 0}, upload=503371]: Done uploading piece. err: <nil>
Dec 29 13:39:29.976135489 (574h10m29.847218524s) [noise=true , address=45.91.116.9:4263     , piece_id=BLFARF6WFKLUW3Y4OZBTYQMNECLFBVLWNOHANH3YQ7F3G3OVL3FA, seg_pos={0 0}, upload=503371]: Done uploading piece. err: <nil>
Dec 29 13:39:30.001624723 (574h10m29.872707788s) [noise=true , address=45.91.116.8:4045     , piece_id=ADUH2TEBPHYJGWPS7ANWHTSYZPPPDAY2HETVIQXVF4W2FXU742CA, seg_pos={0 0}, upload=503371]: Done uploading piece. err: <nil>
Dec 29 13:39:30.011863019 (574h10m29.882946064s) [noise=true , address=45.91.116.11:4117    , piece_id=XEA6MMRD5C7FQXZMBYRBS3PKUVCLGH66JBRN57YUXLQVCYXM6YXA, seg_pos={0 0}, upload=503371]: Done uploading piece. err: <nil>
Dec 29 13:39:30.244682402 (574h10m30.115765437s) [noise=true , address=77.90.31.17:4773     , piece_id=7QLG4MWCALBSVPCMSV75DRR2H3DIEWLISYWASSZO7IXPLBPQQG5A, seg_pos={0 0}, upload=503371]: Done uploading piece. err: <nil>
Dec 29 13:39:30.244720124 (574h10m30.115803169s) [seg_pos={0 0}, upload=503371]: Segment reached optimal threshold of 13 pieces.
Dec 29 13:39:30.244953799 (574h10m30.116036864s) [noise=true , address=145.79.227.44:4193   , piece_id=CLIAY5DZK3EP6Q7MC5OWGNF27FWPXFDHCA2VKFVUWT6L53RQG4IA, seg_pos={0 0}, upload=503371]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF
Dec 29 13:39:30.244963066 (574h10m30.116046101s) [noise=true , address=145.79.227.44:4193   , piece_id=CLIAY5DZK3EP6Q7MC5OWGNF27FWPXFDHCA2VKFVUWT6L53RQG4IA, seg_pos={0 0}, upload=503371]: Long tail cancellation due to optimal threshold
Dec 29 13:39:30.245039031 (574h10m30.116122066s) [noise=true , address=45.91.116.13:4154    , piece_id=VPGRYHVJGCOVRB2RI4P3T7V5WLFLQ452IABYGJR2RC2LXTKDXUJA, seg_pos={0 0}, upload=503371]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: write tcp 188.166.225.170:38266->45.91.116.13:4154: use of closed network connection; write tcp 188.166.225.170:38266->45.91.116.13:4154: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38266->45.91.116.13:4154: use of closed network connection
Dec 29 13:39:30.245055112 (574h10m30.116138147s) [noise=true , address=45.91.116.13:4154    , piece_id=VPGRYHVJGCOVRB2RI4P3T7V5WLFLQ452IABYGJR2RC2LXTKDXUJA, seg_pos={0 0}, upload=503371]: Long tail cancellation due to optimal threshold
Dec 29 13:39:30.245091711 (574h10m30.116174746s) [noise=true , address=77.90.31.19:4841     , piece_id=NNGE6HDMWUG2FBAPKCNSZJWWJTMSHRYMETXB45GCF5ENV5NPBIAQ, seg_pos={0 0}, upload=503371]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 188.166.225.170:35534->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:35534->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35534->77.90.31.19:4841: use of closed network connection
Dec 29 13:39:30.245097973 (574h10m30.116181008s) [noise=true , address=77.90.31.19:4841     , piece_id=NNGE6HDMWUG2FBAPKCNSZJWWJTMSHRYMETXB45GCF5ENV5NPBIAQ, seg_pos={0 0}, upload=503371]: Long tail cancellation due to optimal threshold
Dec 29 13:39:30.245116719 (574h10m30.116199754s) [seg_pos={0 0}, upload=503371]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 13:39:30.245195188 (574h10m30.116278224s) [upload=503371]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 13:39:30.245202673 (574h10m30.116285708s) [upload=503371]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 13:39:30.548923981 (574h10m30.420007016s) [upload=503371]: Done uploading segments...
Dec 29 13:39:30.548942808 (574h10m30.420025843s) [upload=503371]: Upload finished. err: <nil>
Duration: 30.003407717s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
54 out of 60 is failed (threshold=6) [XXX.XX.XXXXXXX.XX.XXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXX.XXXXXX]
duration: 
5.001475023s
trace: 
https://jaeger.dev.storj.io/trace/1c3e916aeeee97f2
FAIL (2025-12-29 13:39:54 UTC) 
orbiter warning: context deadline exceeded
Duration: 5.001502495s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [.....X.....................XX...............................]
duration: 
40.983031377s
trace: 
https://jaeger.dev.storj.io/trace/77d26141ea11d228
FAIL (2025-12-29 13:39:11 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12J2weU7Bmsr3RdQQ9QMuadT96XReBkyxTeaLJ2SEqVGGFX1PWu, Piece ID: JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ) context deadline exceeded; piecestore: (Node ID: 125sm6iHV94mJFLWrKeyRMH4b1qSMo32nDuW8uQr71xUL7Cema2, Piece ID: INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA) context deadline exceeded; piecestore: (Node ID: 12i57FF97oEddYLZpNxSq5or9WLEquL8ZW1cVUuW4AThBbDqcCL, Piece ID: Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA) context deadline exceeded
======LOGS======
Q2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.692912334 (573h43m3.563995379s) [noise=true , address=77.90.31.16:4759     , piece_id=ARC4HRP57U5VQXZTT63QYLXBPZURI5VMJBIQZOFKGLJBQ3K3KGLQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.693322595 (573h43m3.56440564s) [noise=true , address=77.90.31.15:4711     , piece_id=JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.693739649 (573h43m3.564822694s) [noise=true , address=45.91.116.6:4981     , piece_id=INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.694315887 (573h43m3.565398942s) [noise=true , address=145.79.227.4:4220    , piece_id=SHHJ7EW7DTNOZXVBP6A64VIX7P6AQN62TBCUT3KTH4MIB6UXY3BA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.694765222 (573h43m3.565848267s) [noise=true , address=145.79.227.29:4952   , piece_id=DKYDB2R4DNCGH6T3SLRR2MOZZUPVZZVRL2BKGYOBYXJJVPSZEJ4Q, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.695202545 (573h43m3.56628559s) [noise=true , address=145.79.227.40:4058   , piece_id=KRHH2QZKSDPWTSSKYDVDWGKFJD3LPGP6U7KI2PPJ5T3TXYIUGJWA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.695687528 (573h43m3.566770584s) [noise=true , address=77.90.31.19:4857     , piece_id=Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.696135221 (573h43m3.567218276s) [noise=true , address=45.91.116.8:4051     , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.696703733 (573h43m3.567786778s) [noise=true , address=64.49.13.170:4720    , piece_id=HF6X3OT2EPPDE3DWAPWABVZUR3ERDBWXHG3NGNKEWZAD7G5AN4TA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.698418298 (573h43m3.569501343s) [noise=true , address=145.79.227.36:4520   , piece_id=MYJSXUH563JFUW5Q7YUIWL7466G6ASY5WXR3T2PGUFFCTRTOBYJA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.700198829 (573h43m3.571281874s) [noise=true , address=77.90.31.18:4821     , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.701993145 (573h43m3.57307619s) [noise=true , address=193.149.164.143:4699 , piece_id=UDFDCYO25MOJBTMDE6F2WILXE3HLOB75UJZ4LW7ZMNSCOBJI6UJA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.703884687 (573h43m3.574967732s) [noise=true , address=145.79.227.33:4893   , piece_id=XGLX3DFNAY4F4GPGCGZ4ONDJ7NK62SAKX3PBRWHWI74EF3ZFMUSA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.705687109 (573h43m3.576770154s) [noise=true , address=45.91.116.10:4094    , piece_id=6UUIZNTESBBWJMDQ5ZTCHU7S6YKM4GDKO4ZAC6T2FUTPVGFFDS3A, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.707557651 (573h43m3.578640696s) [noise=true , address=145.79.227.43:4164   , piece_id=J6Q4ZDOIIIBA2FXGZZTD6AYS7KGAWSRXXUBAPDPQWYKNU5GNXBBQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:05.168247719 (573h43m5.039330754s) [upload=502975]: Next returned nil segment
Dec 29 13:12:05.168259962 (573h43m5.039343007s) [upload=502975]: Waiting for error group managing segments...
Dec 29 13:12:05.558873127 (573h43m5.429956162s) [noise=true , address=145.79.227.4:4220    , piece_id=SHHJ7EW7DTNOZXVBP6A64VIX7P6AQN62TBCUT3KTH4MIB6UXY3BA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.563024312 (573h43m5.434107347s) [noise=true , address=64.49.13.170:4720    , piece_id=HF6X3OT2EPPDE3DWAPWABVZUR3ERDBWXHG3NGNKEWZAD7G5AN4TA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.564115719 (573h43m5.435198755s) [noise=true , address=145.79.227.36:4520   , piece_id=MYJSXUH563JFUW5Q7YUIWL7466G6ASY5WXR3T2PGUFFCTRTOBYJA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.593873402 (573h43m5.464956447s) [noise=true , address=145.79.227.29:4952   , piece_id=DKYDB2R4DNCGH6T3SLRR2MOZZUPVZZVRL2BKGYOBYXJJVPSZEJ4Q, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.898194604 (573h43m5.769277639s) [noise=true , address=45.91.116.10:4094    , piece_id=6UUIZNTESBBWJMDQ5ZTCHU7S6YKM4GDKO4ZAC6T2FUTPVGFFDS3A, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.912845292 (573h43m5.783928337s) [noise=true , address=77.90.31.19:4857     , piece_id=Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.426532207 (573h43m6.297615252s) [noise=true , address=193.149.164.143:4699 , piece_id=UDFDCYO25MOJBTMDE6F2WILXE3HLOB75UJZ4LW7ZMNSCOBJI6UJA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.449589562 (573h43m6.320672607s) [noise=true , address=77.90.31.15:4711     , piece_id=JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.453186841 (573h43m6.324269876s) [noise=true , address=145.79.227.40:4058   , piece_id=KRHH2QZKSDPWTSSKYDVDWGKFJD3LPGP6U7KI2PPJ5T3TXYIUGJWA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.510769276 (573h43m6.381852321s) [noise=true , address=145.79.227.43:4164   , piece_id=J6Q4ZDOIIIBA2FXGZZTD6AYS7KGAWSRXXUBAPDPQWYKNU5GNXBBQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.518868879 (573h43m6.389951924s) [noise=true , address=145.79.227.33:4893   , piece_id=XGLX3DFNAY4F4GPGCGZ4ONDJ7NK62SAKX3PBRWHWI74EF3ZFMUSA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.519238043 (573h43m6.390321098s) [noise=true , address=45.91.116.6:4981     , piece_id=INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.523211328 (573h43m6.394294373s) [noise=true , address=77.90.31.16:4759     , piece_id=ARC4HRP57U5VQXZTT63QYLXBPZURI5VMJBIQZOFKGLJBQ3K3KGLQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.767869783 (573h43m6.638952818s) [noise=true , address=145.79.227.34:4924   , piece_id=MAQXQTAGXQ7AEPUCESUSUGMHSDMMXS7HHKAEA7B3HW5X66PDUF5A, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.767889431 (573h43m6.638972476s) [seg_pos={0 0}, upload=502975]: Segment reached optimal threshold of 14 pieces.
Dec 29 13:12:06.768102116 (573h43m6.639185161s) [noise=true , address=45.91.116.8:4051     , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1preT33ei4yVXXYVLHhaUbHbiyL14oHFEiof6cCWPig83W6CT3): protocol: expected piece hash; context canceled; EOF
Dec 29 13:12:06.768108257 (573h43m6.639191302s) [noise=true , address=45.91.116.8:4051     , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768219208 (573h43m6.639302254s) [noise=true , address=77.90.31.18:4821     , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH): protocol: expected piece hash; context canceled; EOF
Dec 29 13:12:06.768224789 (573h43m6.639307824s) [noise=true , address=77.90.31.18:4821     , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768384213 (573h43m6.639467258s) [noise=true , address=64.49.13.169:4711    , piece_id=4F4T4YKXMT7NF2C5KBUGJHQJGZL4VCKNKYB266PCVI3UACHJXZ5Q, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection
Dec 29 13:12:06.768399261 (573h43m6.639482296s) [noise=true , address=64.49.13.169:4711    , piece_id=4F4T4YKXMT7NF2C5KBUGJHQJGZL4VCKNKYB266PCVI3UACHJXZ5Q, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768732636 (573h43m6.639815671s) [noise=true , address=64.49.13.169:4709    , piece_id=TBHHX5ES2A6NQRGP25HIC7J7IBWLOJGQ2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:35310->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:35310->64.49.13.169:4709: use of closed network connection
Dec 29 13:12:06.768745460 (573h43m6.639828505s) [noise=true , address=64.49.13.169:4709    , piece_id=TBHHX5ES2A6NQRGP25HIC7J7IBWLOJGQ2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768784586 (573h43m6.639867631s) [seg_pos={0 0}, upload=502975]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 13:12:06.768880217 (573h43m6.639963252s) [upload=502975]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 13:12:06.768884906 (573h43m6.639967941s) [upload=502975]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 13:12:07.067188330 (573h43m6.938271365s) [upload=502975]: Done uploading segments...
Dec 29 13:12:07.067251099 (573h43m6.938334134s) [upload=502975]: Upload finished. err: <nil>
Duration: 40.98304821s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
53 out of 60 is failed (threshold=6) [.XXX.XX.XXXXXXX.XX.XXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXX.XXXXX]
duration: 
5.000960995s
trace: 
https://jaeger.dev.storj.io/trace/58a2f357b4d4ada4
FAIL (2025-12-29 13:39:01 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000984178s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
6 out of 60 is failed (threshold=3) [.......X....X.X....X.X....................................X.]
duration: 
20.536078878s
trace: 
https://jaeger.dev.storj.io/trace/1f23099550b2a511
FAIL (2025-12-29 13:38:58 UTC) 
orbiter warning: context deadline exceeded
Duration: 20.536169341s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
31 out of 60 is failed (threshold=3) [X.X.XXX.X.X.X.XX..X.XXX.XX..X.XX..XXX...X..X...X...X.XXXX.X.]
duration: 
5.892320619s
trace: 
https://jaeger.dev.storj.io/trace/6204d82e10320b11
FAIL (2025-12-29 13:38:51 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: AOGAVQDI5JISOYLKFV2U33F4NAOUVHYWKSRGSZ6KXWLD2MO36DYA) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: YC5PG65MPKZ2C2NFTKVIZYNZNXIZGDZ3HTZ7X45M3L6COEUPYQOA) context deadline exceeded
======LOGS======
Dec 29 13:38:16.080217687 (574h9m15.951300722s) [upload=503350]: Starting upload
Dec 29 13:38:16.080373565 (574h9m15.95145661s) [upload=503350]: Done starting upload
Dec 29 13:38:16.080425713 (574h9m15.951508749s) [upload=503350]: Uploading segments...
Dec 29 13:38:16.080575600 (574h9m15.951658645s) [upload=503350]: Got next segment. Inline: false
Dec 29 13:38:16.080623200 (574h9m15.951706245s) [upload=503350]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 13:38:16.080628860 (574h9m15.951711926s) [upload=503350]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 13:38:16.331754721 (574h9m16.202837767s) [seg_pos={0 0}, upload=503350]: Begin upload segment...
Dec 29 13:38:16.331813093 (574h9m16.202896128s) [seg_pos={0 0}, upload=503350]: Done begin upload segment.
Dec 29 13:38:16.331880642 (574h9m16.202963707s) [noise=true , address=145.79.227.16:4910   , piece_id=44QDN6IXX5BSDQ7RDVEOUHNBFR4MWNWGXMKYV23P277FREAZCHKQ, seg_pos={0 0}, upload=503350]: Uploading piece...
Dec 29 13:38:16.331903815 (574h9m16.20298686s) [noise=true , address=145.79.227.32:4834   , piece_id=MGFDE7IWADWKAJVUGMBLMR3WVOP3OYOSTP34M2XPQHDQICQG5Q5Q, seg_pos={0 0}, upload=503350]: Uploading piece...
Dec 29 13:38:16.332499810 (574h9m16.203582855s) [noise=true , address=145.79.227.46:4602   , piece_id=5WGQS5FDQOIOSNCRLUTSJ5EAHHUTCUJGL3OF7S7WAGGX3OM6IFXQ, seg_pos={0 0}, upload=503350]: Uploading piece...
Dec 29 13:38:16.333134629 (574h9m16.204217664s) [noise=true , address=45.91.116.11:4119    , piece_id=F52IRNIPAHL7ANK2TGOH6KD7K3QS6QDDTES4ZZ7V53R3EOTQ4TSQ, seg_pos={0 0}, upload=503350]: Uploading piece...
Dec 29 13:38:16.334503606 (574h9m16.205586641s) [noise=true , address=145.79.227.36:4519   , piece_id=3IKJUFJX3AEP2LYENDNLZJ67UQO23SFNS6BBCKIYIBQIHY3HJATA, seg_pos={0 0}, upload=503350]: Uploading piece...
Dec 29 13:38:16.334916232 (574h9m16.205999267s) [noise=true , address=145.79.227.24:4582   , piece_id=5SO7D262GNHMU663FDLBJTR65ZRYV2WKG7YFABM6E74JSLRYLUUQ, seg_pos={0 0}, upload=503350]: Uploading piece...
Dec 29 13:38:16.335798101 (574h9m16.206881136s) [noise=true , address=145.79.227.52:4445   , piece_id=SBX3XFQM4HRFGMWORPDGDMLB5TTRG7R7F7KPDINNWW25YXD5RIIA, seg_pos={0 0}, upload=503350]: Uploading piece...
Dec 29 13:38:16.336058888 (574h9m16.207141923s) [noise=true , address=45.91.116.7:4009     , piece_id=AOGAVQDI5JISOYLKFV2U33F4NAOUVHYWKSRGSZ6KXWLD2MO36DYA, seg_pos={0 0}, upload=503350]: Uploading piece...
Dec 29 13:38:16.336978098 (574h9m16.208061133s) [noise=true , address=45.91.116.8:4045     , piece_id=IY4YBSUBLJ6H24GX3JEK4B6WZZQAVO76RIX7MJKHMDEPRRW7QTTQ, seg_pos={0 0}, upload=503350]: Uploading piece...
Dec 29 13:38:16.337302465 (574h9m16.2083855s) [noise=true , address=145.79.227.34:4905   , piece_id=NGQAL3SZEG6QGHD6OY5NNNTNUAKJYEJFZKPH6W3DTC4TOG55DCDA, seg_pos={0 0}, upload=503350]: Uploading piece...
Dec 29 13:38:16.338835324 (574h9m16.209918359s) [noise=true , address=145.79.227.37:4567   , piece_id=ETDHU7HL3CJZWOEFWAWYUQPQ73EMIW7GBONFP5RZYIJ5NBJIEZWA, seg_pos={0 0}, upload=503350]: Uploading piece...
Dec 29 13:38:16.340593001 (574h9m16.211676046s) [noise=true , address=77.90.31.11:4554     , piece_id=PAVHHBI65AK34JWR7CBZJTRFJORBKIOZ2JMC65MCGXUWWWLPJARQ, seg_pos={0 0}, upload=503350]: Uploading piece...
Dec 29 13:38:16.342715101 (574h9m16.213798136s) [noise=true , address=145.79.227.39:4014   , piece_id=DH2KXVSDYOCXCX6QMLFKE4R225RVIFJDTTAAIDVSQNU2SKPDQL4A, seg_pos={0 0}, upload=503350]: Uploading piece...
Dec 29 13:38:16.343899637 (574h9m16.214982682s) [noise=true , address=45.91.116.10:4082    , piece_id=YC5PG65MPKZ2C2NFTKVIZYNZNXIZGDZ3HTZ7X45M3L6COEUPYQOA, seg_pos={0 0}, upload=503350]: Uploading piece...
Dec 29 13:38:16.344299278 (574h9m16.215382313s) [noise=true , address=145.79.227.38:4979   , piece_id=JPTVJOE7HMSJKVU4OTFNOUH655ODKUBYUWDWGQJYRJZAWJ3P73RQ, seg_pos={0 0}, upload=503350]: Uploading piece...
Dec 29 13:38:16.345469276 (574h9m16.216552311s) [noise=true , address=145.79.227.2:4115    , piece_id=JLYV6GRGCLO7J7BHTTBF7NLEGMUQFDYCCT42ODMAZ6KGCH4BP36A, seg_pos={0 0}, upload=503350]: Uploading piece...
Dec 29 13:38:17.976860425 (574h9m17.84794346s) [upload=503350]: Next returned nil segment
Dec 29 13:38:17.976873580 (574h9m17.847956615s) [upload=503350]: Waiting for error group managing segments...
Dec 29 13:38:18.313430053 (574h9m18.184513098s) [noise=true , address=145.79.227.37:4567   , piece_id=ETDHU7HL3CJZWOEFWAWYUQPQ73EMIW7GBONFP5RZYIJ5NBJIEZWA, seg_pos={0 0}, upload=503350]: Done uploading piece. err: <nil>
Dec 29 13:38:18.324724369 (574h9m18.195807404s) [noise=true , address=145.79.227.32:4834   , piece_id=MGFDE7IWADWKAJVUGMBLMR3WVOP3OYOSTP34M2XPQHDQICQG5Q5Q, seg_pos={0 0}, upload=503350]: Done uploading piece. err: <nil>
Dec 29 13:38:18.327051590 (574h9m18.198134625s) [noise=true , address=145.79.227.52:4445   , piece_id=SBX3XFQM4HRFGMWORPDGDMLB5TTRG7R7F7KPDINNWW25YXD5RIIA, seg_pos={0 0}, upload=503350]: Done uploading piece. err: <nil>
Dec 29 13:38:18.327369306 (574h9m18.198452341s) [noise=true , address=145.79.227.16:4910   , piece_id=44QDN6IXX5BSDQ7RDVEOUHNBFR4MWNWGXMKYV23P277FREAZCHKQ, seg_pos={0 0}, upload=503350]: Done uploading piece. err: <nil>
Dec 29 13:38:18.328609346 (574h9m18.199692381s) [noise=true , address=145.79.227.46:4602   , piece_id=5WGQS5FDQOIOSNCRLUTSJ5EAHHUTCUJGL3OF7S7WAGGX3OM6IFXQ, seg_pos={0 0}, upload=503350]: Done uploading piece. err: <nil>
Dec 29 13:38:18.330843430 (574h9m18.201926465s) [noise=true , address=145.79.227.24:4582   , piece_id=5SO7D262GNHMU663FDLBJTR65ZRYV2WKG7YFABM6E74JSLRYLUUQ, seg_pos={0 0}, upload=503350]: Done uploading piece. err: <nil>
Dec 29 13:38:18.338730148 (574h9m18.209813203s) [noise=true , address=145.79.227.36:4519   , piece_id=3IKJUFJX3AEP2LYENDNLZJ67UQO23SFNS6BBCKIYIBQIHY3HJATA, seg_pos={0 0}, upload=503350]: Done uploading piece. err: <nil>
Dec 29 13:38:18.351414631 (574h9m18.222497666s) [noise=true , address=145.79.227.39:4014   , piece_id=DH2KXVSDYOCXCX6QMLFKE4R225RVIFJDTTAAIDVSQNU2SKPDQL4A, seg_pos={0 0}, upload=503350]: Done uploading piece. err: <nil>
Dec 29 13:38:18.352404746 (574h9m18.223487781s) [noise=true , address=145.79.227.38:4979   , piece_id=JPTVJOE7HMSJKVU4OTFNOUH655ODKUBYUWDWGQJYRJZAWJ3P73RQ, seg_pos={0 0}, upload=503350]: Done uploading piece. err: <nil>
Dec 29 13:38:18.353982792 (574h9m18.225065837s) [noise=true , address=145.79.227.2:4115    , piece_id=JLYV6GRGCLO7J7BHTTBF7NLEGMUQFDYCCT42ODMAZ6KGCH4BP36A, seg_pos={0 0}, upload=503350]: Done uploading piece. err: <nil>
Dec 29 13:38:18.656492234 (574h9m18.527575269s) [noise=true , address=77.90.31.11:4554     , piece_id=PAVHHBI65AK34JWR7CBZJTRFJORBKIOZ2JMC65MCGXUWWWLPJARQ, seg_pos={0 0}, upload=503350]: Done uploading piece. err: <nil>
Dec 29 13:38:18.791906142 (574h9m18.662989177s) [noise=true , address=45.91.116.7:4009     , piece_id=AOGAVQDI5JISOYLKFV2U33F4NAOUVHYWKSRGSZ6KXWLD2MO36DYA, seg_pos={0 0}, upload=503350]: Done uploading piece. err: <nil>
Dec 29 13:38:19.597306522 (574h9m19.468389557s) [noise=true , address=45.91.116.10:4082    , piece_id=YC5PG65MPKZ2C2NFTKVIZYNZNXIZGDZ3HTZ7X45M3L6COEUPYQOA, seg_pos={0 0}, upload=503350]: Done uploading piece. err: <nil>
Dec 29 13:38:19.597329747 (574h9m19.468412782s) [seg_pos={0 0}, upload=503350]: Segment reached optimal threshold of 13 pieces.
Dec 29 13:38:19.597553853 (574h9m19.468636888s) [noise=true , address=45.91.116.11:4119    , piece_id=F52IRNIPAHL7ANK2TGOH6KD7K3QS6QDDTES4ZZ7V53R3EOTQ4TSQ, seg_pos={0 0}, upload=503350]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF
Dec 29 13:38:19.597561699 (574h9m19.468644744s) [noise=true , address=45.91.116.11:4119    , piece_id=F52IRNIPAHL7ANK2TGOH6KD7K3QS6QDDTES4ZZ7V53R3EOTQ4TSQ, seg_pos={0 0}, upload=503350]: Long tail cancellation due to optimal threshold
Dec 29 13:38:19.597760557 (574h9m19.468843622s) [noise=true , address=45.91.116.8:4045     , piece_id=IY4YBSUBLJ6H24GX3JEK4B6WZZQAVO76RIX7MJKHMDEPRRW7QTTQ, seg_pos={0 0}, upload=503350]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF
Dec 29 13:38:19.597769453 (574h9m19.468852499s) [noise=true , address=45.91.116.8:4045     , piece_id=IY4YBSUBLJ6H24GX3JEK4B6WZZQAVO76RIX7MJKHMDEPRRW7QTTQ, seg_pos={0 0}, upload=503350]: Long tail cancellation due to optimal threshold
Dec 29 13:38:19.597786967 (574h9m19.468870022s) [noise=true , address=145.79.227.34:4905   , piece_id=NGQAL3SZEG6QGHD6OY5NNNTNUAKJYEJFZKPH6W3DTC4TOG55DCDA, seg_pos={0 0}, upload=503350]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF
Dec 29 13:38:19.597829197 (574h9m19.468914146s) [noise=true , address=145.79.227.34:4905   , piece_id=NGQAL3SZEG6QGHD6OY5NNNTNUAKJYEJFZKPH6W3DTC4TOG55DCDA, seg_pos={0 0}, upload=503350]: Long tail cancellation due to optimal threshold
Dec 29 13:38:19.597901535 (574h9m19.46898458s) [seg_pos={0 0}, upload=503350]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 13:38:19.598030611 (574h9m19.469113646s) [upload=503350]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 13:38:19.598036893 (574h9m19.469119928s) [upload=503350]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 13:38:19.899943928 (574h9m19.771026973s) [upload=503350]: Done uploading segments...
Dec 29 13:38:19.899964267 (574h9m19.771047313s) [upload=503350]: Upload finished. err: <nil>
Duration: 5.892346439s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
31 out of 60 is failed (threshold=3) [.X.X.XXX.X.X.X.XX..X.XXX.XX..X.XX..XXX...X..X...X...X.XXXX.X]
duration: 
30.002892747s
trace: 
https://jaeger.dev.storj.io/trace/6204d82e10320b11
FAIL (2025-12-29 13:38:45 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: AOGAVQDI5JISOYLKFV2U33F4NAOUVHYWKSRGSZ6KXWLD2MO36DYA) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: YC5PG65MPKZ2C2NFTKVIZYNZNXIZGDZ3HTZ7X45M3L6COEUPYQOA) context deadline exceeded
======LOGS======
Dec 29 13:38:16.080217687 (574h9m15.951300722s) [upload=503350]: Starting upload
Dec 29 13:38:16.080373565 (574h9m15.95145661s) [upload=503350]: Done starting upload
Dec 29 13:38:16.080425713 (574h9m15.951508749s) [upload=503350]: Uploading segments...
Dec 29 13:38:16.080575600 (574h9m15.951658645s) [upload=503350]: Got next segment. Inline: false
Dec 29 13:38:16.080623200 (574h9m15.951706245s) [upload=503350]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 13:38:16.080628860 (574h9m15.951711926s) [upload=503350]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 13:38:16.331754721 (574h9m16.202837767s) [seg_pos={0 0}, upload=503350]: Begin upload segment...
Dec 29 13:38:16.331813093 (574h9m16.202896128s) [seg_pos={0 0}, upload=503350]: Done begin upload segment.
Dec 29 13:38:16.331880642 (574h9m16.202963707s) [noise=true , address=145.79.227.16:4910   , piece_id=44QDN6IXX5BSDQ7RDVEOUHNBFR4MWNWGXMKYV23P277FREAZCHKQ, seg_pos={0 0}, upload=503350]: Uploading piece...
Dec 29 13:38:16.331903815 (574h9m16.20298686s) [noise=true , address=145.79.227.32:4834   , piece_id=MGFDE7IWADWKAJVUGMBLMR3WVOP3OYOSTP34M2XPQHDQICQG5Q5Q, seg_pos={0 0}, upload=503350]: Uploading piece...
Dec 29 13:38:16.332499810 (574h9m16.203582855s) [noise=true , address=145.79.227.46:4602   , piece_id=5WGQS5FDQOIOSNCRLUTSJ5EAHHUTCUJGL3OF7S7WAGGX3OM6IFXQ, seg_pos={0 0}, upload=503350]: Uploading piece...
Dec 29 13:38:16.333134629 (574h9m16.204217664s) [noise=true , address=45.91.116.11:4119    , piece_id=F52IRNIPAHL7ANK2TGOH6KD7K3QS6QDDTES4ZZ7V53R3EOTQ4TSQ, seg_pos={0 0}, upload=503350]: Uploading piece...
Dec 29 13:38:16.334503606 (574h9m16.205586641s) [noise=true , address=145.79.227.36:4519   , piece_id=3IKJUFJX3AEP2LYENDNLZJ67UQO23SFNS6BBCKIYIBQIHY3HJATA, seg_pos={0 0}, upload=503350]: Uploading piece...
Dec 29 13:38:16.334916232 (574h9m16.205999267s) [noise=true , address=145.79.227.24:4582   , piece_id=5SO7D262GNHMU663FDLBJTR65ZRYV2WKG7YFABM6E74JSLRYLUUQ, seg_pos={0 0}, upload=503350]: Uploading piece...
Dec 29 13:38:16.335798101 (574h9m16.206881136s) [noise=true , address=145.79.227.52:4445   , piece_id=SBX3XFQM4HRFGMWORPDGDMLB5TTRG7R7F7KPDINNWW25YXD5RIIA, seg_pos={0 0}, upload=503350]: Uploading piece...
Dec 29 13:38:16.336058888 (574h9m16.207141923s) [noise=true , address=45.91.116.7:4009     , piece_id=AOGAVQDI5JISOYLKFV2U33F4NAOUVHYWKSRGSZ6KXWLD2MO36DYA, seg_pos={0 0}, upload=503350]: Uploading piece...
Dec 29 13:38:16.336978098 (574h9m16.208061133s) [noise=true , address=45.91.116.8:4045     , piece_id=IY4YBSUBLJ6H24GX3JEK4B6WZZQAVO76RIX7MJKHMDEPRRW7QTTQ, seg_pos={0 0}, upload=503350]: Uploading piece...
Dec 29 13:38:16.337302465 (574h9m16.2083855s) [noise=true , address=145.79.227.34:4905   , piece_id=NGQAL3SZEG6QGHD6OY5NNNTNUAKJYEJFZKPH6W3DTC4TOG55DCDA, seg_pos={0 0}, upload=503350]: Uploading piece...
Dec 29 13:38:16.338835324 (574h9m16.209918359s) [noise=true , address=145.79.227.37:4567   , piece_id=ETDHU7HL3CJZWOEFWAWYUQPQ73EMIW7GBONFP5RZYIJ5NBJIEZWA, seg_pos={0 0}, upload=503350]: Uploading piece...
Dec 29 13:38:16.340593001 (574h9m16.211676046s) [noise=true , address=77.90.31.11:4554     , piece_id=PAVHHBI65AK34JWR7CBZJTRFJORBKIOZ2JMC65MCGXUWWWLPJARQ, seg_pos={0 0}, upload=503350]: Uploading piece...
Dec 29 13:38:16.342715101 (574h9m16.213798136s) [noise=true , address=145.79.227.39:4014   , piece_id=DH2KXVSDYOCXCX6QMLFKE4R225RVIFJDTTAAIDVSQNU2SKPDQL4A, seg_pos={0 0}, upload=503350]: Uploading piece...
Dec 29 13:38:16.343899637 (574h9m16.214982682s) [noise=true , address=45.91.116.10:4082    , piece_id=YC5PG65MPKZ2C2NFTKVIZYNZNXIZGDZ3HTZ7X45M3L6COEUPYQOA, seg_pos={0 0}, upload=503350]: Uploading piece...
Dec 29 13:38:16.344299278 (574h9m16.215382313s) [noise=true , address=145.79.227.38:4979   , piece_id=JPTVJOE7HMSJKVU4OTFNOUH655ODKUBYUWDWGQJYRJZAWJ3P73RQ, seg_pos={0 0}, upload=503350]: Uploading piece...
Dec 29 13:38:16.345469276 (574h9m16.216552311s) [noise=true , address=145.79.227.2:4115    , piece_id=JLYV6GRGCLO7J7BHTTBF7NLEGMUQFDYCCT42ODMAZ6KGCH4BP36A, seg_pos={0 0}, upload=503350]: Uploading piece...
Dec 29 13:38:17.976860425 (574h9m17.84794346s) [upload=503350]: Next returned nil segment
Dec 29 13:38:17.976873580 (574h9m17.847956615s) [upload=503350]: Waiting for error group managing segments...
Dec 29 13:38:18.313430053 (574h9m18.184513098s) [noise=true , address=145.79.227.37:4567   , piece_id=ETDHU7HL3CJZWOEFWAWYUQPQ73EMIW7GBONFP5RZYIJ5NBJIEZWA, seg_pos={0 0}, upload=503350]: Done uploading piece. err: <nil>
Dec 29 13:38:18.324724369 (574h9m18.195807404s) [noise=true , address=145.79.227.32:4834   , piece_id=MGFDE7IWADWKAJVUGMBLMR3WVOP3OYOSTP34M2XPQHDQICQG5Q5Q, seg_pos={0 0}, upload=503350]: Done uploading piece. err: <nil>
Dec 29 13:38:18.327051590 (574h9m18.198134625s) [noise=true , address=145.79.227.52:4445   , piece_id=SBX3XFQM4HRFGMWORPDGDMLB5TTRG7R7F7KPDINNWW25YXD5RIIA, seg_pos={0 0}, upload=503350]: Done uploading piece. err: <nil>
Dec 29 13:38:18.327369306 (574h9m18.198452341s) [noise=true , address=145.79.227.16:4910   , piece_id=44QDN6IXX5BSDQ7RDVEOUHNBFR4MWNWGXMKYV23P277FREAZCHKQ, seg_pos={0 0}, upload=503350]: Done uploading piece. err: <nil>
Dec 29 13:38:18.328609346 (574h9m18.199692381s) [noise=true , address=145.79.227.46:4602   , piece_id=5WGQS5FDQOIOSNCRLUTSJ5EAHHUTCUJGL3OF7S7WAGGX3OM6IFXQ, seg_pos={0 0}, upload=503350]: Done uploading piece. err: <nil>
Dec 29 13:38:18.330843430 (574h9m18.201926465s) [noise=true , address=145.79.227.24:4582   , piece_id=5SO7D262GNHMU663FDLBJTR65ZRYV2WKG7YFABM6E74JSLRYLUUQ, seg_pos={0 0}, upload=503350]: Done uploading piece. err: <nil>
Dec 29 13:38:18.338730148 (574h9m18.209813203s) [noise=true , address=145.79.227.36:4519   , piece_id=3IKJUFJX3AEP2LYENDNLZJ67UQO23SFNS6BBCKIYIBQIHY3HJATA, seg_pos={0 0}, upload=503350]: Done uploading piece. err: <nil>
Dec 29 13:38:18.351414631 (574h9m18.222497666s) [noise=true , address=145.79.227.39:4014   , piece_id=DH2KXVSDYOCXCX6QMLFKE4R225RVIFJDTTAAIDVSQNU2SKPDQL4A, seg_pos={0 0}, upload=503350]: Done uploading piece. err: <nil>
Dec 29 13:38:18.352404746 (574h9m18.223487781s) [noise=true , address=145.79.227.38:4979   , piece_id=JPTVJOE7HMSJKVU4OTFNOUH655ODKUBYUWDWGQJYRJZAWJ3P73RQ, seg_pos={0 0}, upload=503350]: Done uploading piece. err: <nil>
Dec 29 13:38:18.353982792 (574h9m18.225065837s) [noise=true , address=145.79.227.2:4115    , piece_id=JLYV6GRGCLO7J7BHTTBF7NLEGMUQFDYCCT42ODMAZ6KGCH4BP36A, seg_pos={0 0}, upload=503350]: Done uploading piece. err: <nil>
Dec 29 13:38:18.656492234 (574h9m18.527575269s) [noise=true , address=77.90.31.11:4554     , piece_id=PAVHHBI65AK34JWR7CBZJTRFJORBKIOZ2JMC65MCGXUWWWLPJARQ, seg_pos={0 0}, upload=503350]: Done uploading piece. err: <nil>
Dec 29 13:38:18.791906142 (574h9m18.662989177s) [noise=true , address=45.91.116.7:4009     , piece_id=AOGAVQDI5JISOYLKFV2U33F4NAOUVHYWKSRGSZ6KXWLD2MO36DYA, seg_pos={0 0}, upload=503350]: Done uploading piece. err: <nil>
Dec 29 13:38:19.597306522 (574h9m19.468389557s) [noise=true , address=45.91.116.10:4082    , piece_id=YC5PG65MPKZ2C2NFTKVIZYNZNXIZGDZ3HTZ7X45M3L6COEUPYQOA, seg_pos={0 0}, upload=503350]: Done uploading piece. err: <nil>
Dec 29 13:38:19.597329747 (574h9m19.468412782s) [seg_pos={0 0}, upload=503350]: Segment reached optimal threshold of 13 pieces.
Dec 29 13:38:19.597553853 (574h9m19.468636888s) [noise=true , address=45.91.116.11:4119    , piece_id=F52IRNIPAHL7ANK2TGOH6KD7K3QS6QDDTES4ZZ7V53R3EOTQ4TSQ, seg_pos={0 0}, upload=503350]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF
Dec 29 13:38:19.597561699 (574h9m19.468644744s) [noise=true , address=45.91.116.11:4119    , piece_id=F52IRNIPAHL7ANK2TGOH6KD7K3QS6QDDTES4ZZ7V53R3EOTQ4TSQ, seg_pos={0 0}, upload=503350]: Long tail cancellation due to optimal threshold
Dec 29 13:38:19.597760557 (574h9m19.468843622s) [noise=true , address=45.91.116.8:4045     , piece_id=IY4YBSUBLJ6H24GX3JEK4B6WZZQAVO76RIX7MJKHMDEPRRW7QTTQ, seg_pos={0 0}, upload=503350]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF
Dec 29 13:38:19.597769453 (574h9m19.468852499s) [noise=true , address=45.91.116.8:4045     , piece_id=IY4YBSUBLJ6H24GX3JEK4B6WZZQAVO76RIX7MJKHMDEPRRW7QTTQ, seg_pos={0 0}, upload=503350]: Long tail cancellation due to optimal threshold
Dec 29 13:38:19.597786967 (574h9m19.468870022s) [noise=true , address=145.79.227.34:4905   , piece_id=NGQAL3SZEG6QGHD6OY5NNNTNUAKJYEJFZKPH6W3DTC4TOG55DCDA, seg_pos={0 0}, upload=503350]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF
Dec 29 13:38:19.597829197 (574h9m19.468914146s) [noise=true , address=145.79.227.34:4905   , piece_id=NGQAL3SZEG6QGHD6OY5NNNTNUAKJYEJFZKPH6W3DTC4TOG55DCDA, seg_pos={0 0}, upload=503350]: Long tail cancellation due to optimal threshold
Dec 29 13:38:19.597901535 (574h9m19.46898458s) [seg_pos={0 0}, upload=503350]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 13:38:19.598030611 (574h9m19.469113646s) [upload=503350]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 13:38:19.598036893 (574h9m19.469119928s) [upload=503350]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 13:38:19.899943928 (574h9m19.771026973s) [upload=503350]: Done uploading segments...
Dec 29 13:38:19.899964267 (574h9m19.771047313s) [upload=503350]: Upload finished. err: <nil>
Duration: 30.002915229s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
4 out of 60 is failed (threshold=3) [X.....X.....................XX..............................]
duration: 
24.162961114s
trace: 
https://jaeger.dev.storj.io/trace/77d26141ea11d228
FAIL (2025-12-29 13:38:30 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12J2weU7Bmsr3RdQQ9QMuadT96XReBkyxTeaLJ2SEqVGGFX1PWu, Piece ID: JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ) context deadline exceeded; piecestore: (Node ID: 125sm6iHV94mJFLWrKeyRMH4b1qSMo32nDuW8uQr71xUL7Cema2, Piece ID: INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA) context deadline exceeded; piecestore: (Node ID: 12i57FF97oEddYLZpNxSq5or9WLEquL8ZW1cVUuW4AThBbDqcCL, Piece ID: Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA) context deadline exceeded
======LOGS======
Q2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.692912334 (573h43m3.563995379s) [noise=true , address=77.90.31.16:4759     , piece_id=ARC4HRP57U5VQXZTT63QYLXBPZURI5VMJBIQZOFKGLJBQ3K3KGLQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.693322595 (573h43m3.56440564s) [noise=true , address=77.90.31.15:4711     , piece_id=JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.693739649 (573h43m3.564822694s) [noise=true , address=45.91.116.6:4981     , piece_id=INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.694315887 (573h43m3.565398942s) [noise=true , address=145.79.227.4:4220    , piece_id=SHHJ7EW7DTNOZXVBP6A64VIX7P6AQN62TBCUT3KTH4MIB6UXY3BA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.694765222 (573h43m3.565848267s) [noise=true , address=145.79.227.29:4952   , piece_id=DKYDB2R4DNCGH6T3SLRR2MOZZUPVZZVRL2BKGYOBYXJJVPSZEJ4Q, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.695202545 (573h43m3.56628559s) [noise=true , address=145.79.227.40:4058   , piece_id=KRHH2QZKSDPWTSSKYDVDWGKFJD3LPGP6U7KI2PPJ5T3TXYIUGJWA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.695687528 (573h43m3.566770584s) [noise=true , address=77.90.31.19:4857     , piece_id=Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.696135221 (573h43m3.567218276s) [noise=true , address=45.91.116.8:4051     , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.696703733 (573h43m3.567786778s) [noise=true , address=64.49.13.170:4720    , piece_id=HF6X3OT2EPPDE3DWAPWABVZUR3ERDBWXHG3NGNKEWZAD7G5AN4TA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.698418298 (573h43m3.569501343s) [noise=true , address=145.79.227.36:4520   , piece_id=MYJSXUH563JFUW5Q7YUIWL7466G6ASY5WXR3T2PGUFFCTRTOBYJA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.700198829 (573h43m3.571281874s) [noise=true , address=77.90.31.18:4821     , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.701993145 (573h43m3.57307619s) [noise=true , address=193.149.164.143:4699 , piece_id=UDFDCYO25MOJBTMDE6F2WILXE3HLOB75UJZ4LW7ZMNSCOBJI6UJA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.703884687 (573h43m3.574967732s) [noise=true , address=145.79.227.33:4893   , piece_id=XGLX3DFNAY4F4GPGCGZ4ONDJ7NK62SAKX3PBRWHWI74EF3ZFMUSA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.705687109 (573h43m3.576770154s) [noise=true , address=45.91.116.10:4094    , piece_id=6UUIZNTESBBWJMDQ5ZTCHU7S6YKM4GDKO4ZAC6T2FUTPVGFFDS3A, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.707557651 (573h43m3.578640696s) [noise=true , address=145.79.227.43:4164   , piece_id=J6Q4ZDOIIIBA2FXGZZTD6AYS7KGAWSRXXUBAPDPQWYKNU5GNXBBQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:05.168247719 (573h43m5.039330754s) [upload=502975]: Next returned nil segment
Dec 29 13:12:05.168259962 (573h43m5.039343007s) [upload=502975]: Waiting for error group managing segments...
Dec 29 13:12:05.558873127 (573h43m5.429956162s) [noise=true , address=145.79.227.4:4220    , piece_id=SHHJ7EW7DTNOZXVBP6A64VIX7P6AQN62TBCUT3KTH4MIB6UXY3BA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.563024312 (573h43m5.434107347s) [noise=true , address=64.49.13.170:4720    , piece_id=HF6X3OT2EPPDE3DWAPWABVZUR3ERDBWXHG3NGNKEWZAD7G5AN4TA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.564115719 (573h43m5.435198755s) [noise=true , address=145.79.227.36:4520   , piece_id=MYJSXUH563JFUW5Q7YUIWL7466G6ASY5WXR3T2PGUFFCTRTOBYJA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.593873402 (573h43m5.464956447s) [noise=true , address=145.79.227.29:4952   , piece_id=DKYDB2R4DNCGH6T3SLRR2MOZZUPVZZVRL2BKGYOBYXJJVPSZEJ4Q, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.898194604 (573h43m5.769277639s) [noise=true , address=45.91.116.10:4094    , piece_id=6UUIZNTESBBWJMDQ5ZTCHU7S6YKM4GDKO4ZAC6T2FUTPVGFFDS3A, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.912845292 (573h43m5.783928337s) [noise=true , address=77.90.31.19:4857     , piece_id=Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.426532207 (573h43m6.297615252s) [noise=true , address=193.149.164.143:4699 , piece_id=UDFDCYO25MOJBTMDE6F2WILXE3HLOB75UJZ4LW7ZMNSCOBJI6UJA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.449589562 (573h43m6.320672607s) [noise=true , address=77.90.31.15:4711     , piece_id=JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.453186841 (573h43m6.324269876s) [noise=true , address=145.79.227.40:4058   , piece_id=KRHH2QZKSDPWTSSKYDVDWGKFJD3LPGP6U7KI2PPJ5T3TXYIUGJWA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.510769276 (573h43m6.381852321s) [noise=true , address=145.79.227.43:4164   , piece_id=J6Q4ZDOIIIBA2FXGZZTD6AYS7KGAWSRXXUBAPDPQWYKNU5GNXBBQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.518868879 (573h43m6.389951924s) [noise=true , address=145.79.227.33:4893   , piece_id=XGLX3DFNAY4F4GPGCGZ4ONDJ7NK62SAKX3PBRWHWI74EF3ZFMUSA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.519238043 (573h43m6.390321098s) [noise=true , address=45.91.116.6:4981     , piece_id=INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.523211328 (573h43m6.394294373s) [noise=true , address=77.90.31.16:4759     , piece_id=ARC4HRP57U5VQXZTT63QYLXBPZURI5VMJBIQZOFKGLJBQ3K3KGLQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.767869783 (573h43m6.638952818s) [noise=true , address=145.79.227.34:4924   , piece_id=MAQXQTAGXQ7AEPUCESUSUGMHSDMMXS7HHKAEA7B3HW5X66PDUF5A, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.767889431 (573h43m6.638972476s) [seg_pos={0 0}, upload=502975]: Segment reached optimal threshold of 14 pieces.
Dec 29 13:12:06.768102116 (573h43m6.639185161s) [noise=true , address=45.91.116.8:4051     , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1preT33ei4yVXXYVLHhaUbHbiyL14oHFEiof6cCWPig83W6CT3): protocol: expected piece hash; context canceled; EOF
Dec 29 13:12:06.768108257 (573h43m6.639191302s) [noise=true , address=45.91.116.8:4051     , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768219208 (573h43m6.639302254s) [noise=true , address=77.90.31.18:4821     , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH): protocol: expected piece hash; context canceled; EOF
Dec 29 13:12:06.768224789 (573h43m6.639307824s) [noise=true , address=77.90.31.18:4821     , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768384213 (573h43m6.639467258s) [noise=true , address=64.49.13.169:4711    , piece_id=4F4T4YKXMT7NF2C5KBUGJHQJGZL4VCKNKYB266PCVI3UACHJXZ5Q, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection
Dec 29 13:12:06.768399261 (573h43m6.639482296s) [noise=true , address=64.49.13.169:4711    , piece_id=4F4T4YKXMT7NF2C5KBUGJHQJGZL4VCKNKYB266PCVI3UACHJXZ5Q, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768732636 (573h43m6.639815671s) [noise=true , address=64.49.13.169:4709    , piece_id=TBHHX5ES2A6NQRGP25HIC7J7IBWLOJGQ2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:35310->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:35310->64.49.13.169:4709: use of closed network connection
Dec 29 13:12:06.768745460 (573h43m6.639828505s) [noise=true , address=64.49.13.169:4709    , piece_id=TBHHX5ES2A6NQRGP25HIC7J7IBWLOJGQ2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768784586 (573h43m6.639867631s) [seg_pos={0 0}, upload=502975]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 13:12:06.768880217 (573h43m6.639963252s) [upload=502975]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 13:12:06.768884906 (573h43m6.639967941s) [upload=502975]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 13:12:07.067188330 (573h43m6.938271365s) [upload=502975]: Done uploading segments...
Dec 29 13:12:07.067251099 (573h43m6.938334134s) [upload=502975]: Upload finished. err: <nil>
Duration: 24.16297521s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
6 out of 60 is failed (threshold=3) [........X....X.X....X.X....................................X]
duration: 
30.000643344s
trace: 
https://jaeger.dev.storj.io/trace/1f23099550b2a511
FAIL (2025-12-29 13:38:25 UTC) 
orbiter warning: context deadline exceeded
Duration: 30.000716343s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
4 out of 60 is failed (threshold=3) [.X.....X.....................XX.............................]
duration: 
34.788243636s
trace: 
https://jaeger.dev.storj.io/trace/77d26141ea11d228
FAIL (2025-12-29 13:38:06 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12J2weU7Bmsr3RdQQ9QMuadT96XReBkyxTeaLJ2SEqVGGFX1PWu, Piece ID: JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ) context deadline exceeded; piecestore: (Node ID: 125sm6iHV94mJFLWrKeyRMH4b1qSMo32nDuW8uQr71xUL7Cema2, Piece ID: INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA) context deadline exceeded; piecestore: (Node ID: 12i57FF97oEddYLZpNxSq5or9WLEquL8ZW1cVUuW4AThBbDqcCL, Piece ID: Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA) context deadline exceeded
======LOGS======
Q2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.692912334 (573h43m3.563995379s) [noise=true , address=77.90.31.16:4759     , piece_id=ARC4HRP57U5VQXZTT63QYLXBPZURI5VMJBIQZOFKGLJBQ3K3KGLQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.693322595 (573h43m3.56440564s) [noise=true , address=77.90.31.15:4711     , piece_id=JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.693739649 (573h43m3.564822694s) [noise=true , address=45.91.116.6:4981     , piece_id=INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.694315887 (573h43m3.565398942s) [noise=true , address=145.79.227.4:4220    , piece_id=SHHJ7EW7DTNOZXVBP6A64VIX7P6AQN62TBCUT3KTH4MIB6UXY3BA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.694765222 (573h43m3.565848267s) [noise=true , address=145.79.227.29:4952   , piece_id=DKYDB2R4DNCGH6T3SLRR2MOZZUPVZZVRL2BKGYOBYXJJVPSZEJ4Q, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.695202545 (573h43m3.56628559s) [noise=true , address=145.79.227.40:4058   , piece_id=KRHH2QZKSDPWTSSKYDVDWGKFJD3LPGP6U7KI2PPJ5T3TXYIUGJWA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.695687528 (573h43m3.566770584s) [noise=true , address=77.90.31.19:4857     , piece_id=Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.696135221 (573h43m3.567218276s) [noise=true , address=45.91.116.8:4051     , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.696703733 (573h43m3.567786778s) [noise=true , address=64.49.13.170:4720    , piece_id=HF6X3OT2EPPDE3DWAPWABVZUR3ERDBWXHG3NGNKEWZAD7G5AN4TA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.698418298 (573h43m3.569501343s) [noise=true , address=145.79.227.36:4520   , piece_id=MYJSXUH563JFUW5Q7YUIWL7466G6ASY5WXR3T2PGUFFCTRTOBYJA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.700198829 (573h43m3.571281874s) [noise=true , address=77.90.31.18:4821     , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.701993145 (573h43m3.57307619s) [noise=true , address=193.149.164.143:4699 , piece_id=UDFDCYO25MOJBTMDE6F2WILXE3HLOB75UJZ4LW7ZMNSCOBJI6UJA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.703884687 (573h43m3.574967732s) [noise=true , address=145.79.227.33:4893   , piece_id=XGLX3DFNAY4F4GPGCGZ4ONDJ7NK62SAKX3PBRWHWI74EF3ZFMUSA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.705687109 (573h43m3.576770154s) [noise=true , address=45.91.116.10:4094    , piece_id=6UUIZNTESBBWJMDQ5ZTCHU7S6YKM4GDKO4ZAC6T2FUTPVGFFDS3A, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.707557651 (573h43m3.578640696s) [noise=true , address=145.79.227.43:4164   , piece_id=J6Q4ZDOIIIBA2FXGZZTD6AYS7KGAWSRXXUBAPDPQWYKNU5GNXBBQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:05.168247719 (573h43m5.039330754s) [upload=502975]: Next returned nil segment
Dec 29 13:12:05.168259962 (573h43m5.039343007s) [upload=502975]: Waiting for error group managing segments...
Dec 29 13:12:05.558873127 (573h43m5.429956162s) [noise=true , address=145.79.227.4:4220    , piece_id=SHHJ7EW7DTNOZXVBP6A64VIX7P6AQN62TBCUT3KTH4MIB6UXY3BA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.563024312 (573h43m5.434107347s) [noise=true , address=64.49.13.170:4720    , piece_id=HF6X3OT2EPPDE3DWAPWABVZUR3ERDBWXHG3NGNKEWZAD7G5AN4TA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.564115719 (573h43m5.435198755s) [noise=true , address=145.79.227.36:4520   , piece_id=MYJSXUH563JFUW5Q7YUIWL7466G6ASY5WXR3T2PGUFFCTRTOBYJA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.593873402 (573h43m5.464956447s) [noise=true , address=145.79.227.29:4952   , piece_id=DKYDB2R4DNCGH6T3SLRR2MOZZUPVZZVRL2BKGYOBYXJJVPSZEJ4Q, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.898194604 (573h43m5.769277639s) [noise=true , address=45.91.116.10:4094    , piece_id=6UUIZNTESBBWJMDQ5ZTCHU7S6YKM4GDKO4ZAC6T2FUTPVGFFDS3A, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.912845292 (573h43m5.783928337s) [noise=true , address=77.90.31.19:4857     , piece_id=Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.426532207 (573h43m6.297615252s) [noise=true , address=193.149.164.143:4699 , piece_id=UDFDCYO25MOJBTMDE6F2WILXE3HLOB75UJZ4LW7ZMNSCOBJI6UJA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.449589562 (573h43m6.320672607s) [noise=true , address=77.90.31.15:4711     , piece_id=JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.453186841 (573h43m6.324269876s) [noise=true , address=145.79.227.40:4058   , piece_id=KRHH2QZKSDPWTSSKYDVDWGKFJD3LPGP6U7KI2PPJ5T3TXYIUGJWA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.510769276 (573h43m6.381852321s) [noise=true , address=145.79.227.43:4164   , piece_id=J6Q4ZDOIIIBA2FXGZZTD6AYS7KGAWSRXXUBAPDPQWYKNU5GNXBBQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.518868879 (573h43m6.389951924s) [noise=true , address=145.79.227.33:4893   , piece_id=XGLX3DFNAY4F4GPGCGZ4ONDJ7NK62SAKX3PBRWHWI74EF3ZFMUSA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.519238043 (573h43m6.390321098s) [noise=true , address=45.91.116.6:4981     , piece_id=INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.523211328 (573h43m6.394294373s) [noise=true , address=77.90.31.16:4759     , piece_id=ARC4HRP57U5VQXZTT63QYLXBPZURI5VMJBIQZOFKGLJBQ3K3KGLQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.767869783 (573h43m6.638952818s) [noise=true , address=145.79.227.34:4924   , piece_id=MAQXQTAGXQ7AEPUCESUSUGMHSDMMXS7HHKAEA7B3HW5X66PDUF5A, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.767889431 (573h43m6.638972476s) [seg_pos={0 0}, upload=502975]: Segment reached optimal threshold of 14 pieces.
Dec 29 13:12:06.768102116 (573h43m6.639185161s) [noise=true , address=45.91.116.8:4051     , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1preT33ei4yVXXYVLHhaUbHbiyL14oHFEiof6cCWPig83W6CT3): protocol: expected piece hash; context canceled; EOF
Dec 29 13:12:06.768108257 (573h43m6.639191302s) [noise=true , address=45.91.116.8:4051     , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768219208 (573h43m6.639302254s) [noise=true , address=77.90.31.18:4821     , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH): protocol: expected piece hash; context canceled; EOF
Dec 29 13:12:06.768224789 (573h43m6.639307824s) [noise=true , address=77.90.31.18:4821     , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768384213 (573h43m6.639467258s) [noise=true , address=64.49.13.169:4711    , piece_id=4F4T4YKXMT7NF2C5KBUGJHQJGZL4VCKNKYB266PCVI3UACHJXZ5Q, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection
Dec 29 13:12:06.768399261 (573h43m6.639482296s) [noise=true , address=64.49.13.169:4711    , piece_id=4F4T4YKXMT7NF2C5KBUGJHQJGZL4VCKNKYB266PCVI3UACHJXZ5Q, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768732636 (573h43m6.639815671s) [noise=true , address=64.49.13.169:4709    , piece_id=TBHHX5ES2A6NQRGP25HIC7J7IBWLOJGQ2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:35310->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:35310->64.49.13.169:4709: use of closed network connection
Dec 29 13:12:06.768745460 (573h43m6.639828505s) [noise=true , address=64.49.13.169:4709    , piece_id=TBHHX5ES2A6NQRGP25HIC7J7IBWLOJGQ2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768784586 (573h43m6.639867631s) [seg_pos={0 0}, upload=502975]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 13:12:06.768880217 (573h43m6.639963252s) [upload=502975]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 13:12:06.768884906 (573h43m6.639967941s) [upload=502975]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 13:12:07.067188330 (573h43m6.938271365s) [upload=502975]: Done uploading segments...
Dec 29 13:12:07.067251099 (573h43m6.938334134s) [upload=502975]: Upload finished. err: <nil>
Duration: 34.788263042s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
53 out of 60 is failed (threshold=6) [X.XXX.XX.XXXXXXX.XX.XXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXX.XXXX]
duration: 
5.001583569s
trace: 
https://jaeger.dev.storj.io/trace/3421e4cd74186bfc
FAIL (2025-12-29 13:37:57 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001609278s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
31 out of 60 is failed (threshold=3) [X.X.X.XXX.X.X.X.XX..X.XXX.XX..X.XX..XXX...X..X...X...X.XXXX.]
duration: 
5.726713834s
trace: 
https://jaeger.dev.storj.io/trace/627523b37459ea01
FAIL (2025-12-29 13:37:41 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 29 13:36:46.538550545 (574h7m46.40963358s) [upload=503327]: Starting upload
Dec 29 13:36:46.538649083 (574h7m46.409732118s) [upload=503327]: Done starting upload
Dec 29 13:36:46.538691995 (574h7m46.40977504s) [upload=503327]: Uploading segments...
Dec 29 13:36:46.538933555 (574h7m46.41001659s) [upload=503327]: Got next segment. Inline: false
Dec 29 13:36:46.539704553 (574h7m46.410787588s) [upload=503327]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 13:36:46.539712998 (574h7m46.410796044s) [upload=503327]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 13:36:46.784755246 (574h7m46.655838281s) [seg_pos={0 0}, upload=503327]: Begin upload segment...
Dec 29 13:36:46.784794711 (574h7m46.655877746s) [seg_pos={0 0}, upload=503327]: Done begin upload segment.
Dec 29 13:36:46.784825379 (574h7m46.655908415s) [noise=true , address=45.91.116.5:4299     , piece_id=NFDWU3WN2QCPTV5TZSSMETZMYZ2HTRWKD7I7SDJR7JNSD7FDVKNQ, seg_pos={0 0}, upload=503327]: Uploading piece...
Dec 29 13:36:46.784943334 (574h7m46.656026379s) [noise=true , address=145.79.227.4:4187    , piece_id=XIY2EU3CFR2PHQTD3JTQ6G56W42RGBRRKG2GCBJTHGLPZUH53KPQ, seg_pos={0 0}, upload=503327]: Uploading piece...
Dec 29 13:36:46.785482059 (574h7m46.656565124s) [noise=true , address=145.79.227.36:4519   , piece_id=Q45YJMQ7FILAA76VQS3OKP5LWK5B6HORI2KZY4OVHEBTETRA64WQ, seg_pos={0 0}, upload=503327]: Uploading piece...
Dec 29 13:36:46.785945172 (574h7m46.657028207s) [noise=true , address=77.90.31.19:4842     , piece_id=2UFBA3FKX3LMLYWVCBUN22EYHPFDWQXB4MAYLBDE23BWUHR7LKHQ, seg_pos={0 0}, upload=503327]: Uploading piece...
Dec 29 13:36:46.786352827 (574h7m46.657435862s) [noise=true , address=145.79.227.15:4876   , piece_id=U7N7JRFGUXWGLKBLGGZHIH3KLMLC75QV5QS3FRPVKZLH423BSW2A, seg_pos={0 0}, upload=503327]: Uploading piece...
Dec 29 13:36:46.786638151 (574h7m46.657721186s) [noise=true , address=45.91.116.8:4046     , piece_id=PD77P7625XN6PWI32PMIYIAIWZWA6DEJW5YS5PDUKMOELQHGFYPA, seg_pos={0 0}, upload=503327]: Uploading piece...
Dec 29 13:36:46.787020309 (574h7m46.658103344s) [noise=true , address=145.79.227.50:4373   , piece_id=RBFAMNUMC3IEBTRIXGXDZAFSNSIV4E65NESEST7QBBLJKW46HC3Q, seg_pos={0 0}, upload=503327]: Uploading piece...
Dec 29 13:36:46.787106022 (574h7m46.658189057s) [noise=true , address=77.90.31.14:4663     , piece_id=GTW3LDMT3QKPTSAPSJFV2MKU2XN2DFSIS77ZWZCYOLIM5KVBTNYQ, seg_pos={0 0}, upload=503327]: Uploading piece...
Dec 29 13:36:46.787643516 (574h7m46.658726551s) [noise=true , address=145.79.227.37:4567   , piece_id=LVZXVKKTUFQMSPXVLJU5I5KG6U7DQ457JQP2JSZZE63SIOF6YZ5A, seg_pos={0 0}, upload=503327]: Uploading piece...
Dec 29 13:36:46.787722436 (574h7m46.658805471s) [noise=true , address=145.79.227.39:4013   , piece_id=WZJBAXRNGCYMFI672KOD47CNYL6R2DHMTIRMSBH7ZTM6IAY3JWNQ, seg_pos={0 0}, upload=503327]: Uploading piece...
Dec 29 13:36:46.788257053 (574h7m46.659340088s) [noise=true , address=145.79.227.20:4437   , piece_id=ICZXCCIVIGBUVE24OJDNBDWGAMML2F7S27DCMVENLWV5W5CTXAWA, seg_pos={0 0}, upload=503327]: Uploading piece...
Dec 29 13:36:46.788285327 (574h7m46.659368362s) [noise=true , address=45.91.116.4:4937     , piece_id=HC3UMDTD3Y532PVH4FAZWCCA4YJESCXFRYDLQ6HNNVJO34K34TPA, seg_pos={0 0}, upload=503327]: Uploading piece...
Dec 29 13:36:46.788882504 (574h7m46.659965549s) [noise=true , address=145.79.227.13:4802   , piece_id=YHXPFWFP6GOLTFYW66YCULYPYNUEE5CX2QWW7RBNQQJM7WJTPMGA, seg_pos={0 0}, upload=503327]: Uploading piece...
Dec 29 13:36:46.790211465 (574h7m46.66129451s) [noise=true , address=145.79.227.45:4230   , piece_id=KBYGONO2HPRPZ6WJEC2HRIU3AHPTNDADIYAHKV7G3XQV5QDV4EGA, seg_pos={0 0}, upload=503327]: Uploading piece...
Dec 29 13:36:46.790644229 (574h7m46.661727275s) [noise=true , address=145.79.227.3:4149    , piece_id=KYLZRXN33U5EYWLCTG6IBBTUFMWCT3Y3YCLE65Q7XHSENKJCSZCA, seg_pos={0 0}, upload=503327]: Uploading piece...
Dec 29 13:36:46.791079989 (574h7m46.662163034s) [noise=true , address=45.91.116.11:4118    , piece_id=2W6VDDJ5VEW3VYB32BBCXGEACK3PEVFAUK7QMBW7YX5C3CK2V3MA, seg_pos={0 0}, upload=503327]: Uploading piece...
Dec 29 13:36:48.486010585 (574h7m48.35709362s) [upload=503327]: Next returned nil segment
Dec 29 13:36:48.486028931 (574h7m48.357111976s) [upload=503327]: Waiting for error group managing segments...
Dec 29 13:36:48.768858048 (574h7m48.639941084s) [noise=true , address=145.79.227.3:4149    , piece_id=KYLZRXN33U5EYWLCTG6IBBTUFMWCT3Y3YCLE65Q7XHSENKJCSZCA, seg_pos={0 0}, upload=503327]: Done uploading piece. err: <nil>
Dec 29 13:36:48.770501598 (574h7m48.641584633s) [noise=true , address=145.79.227.13:4802   , piece_id=YHXPFWFP6GOLTFYW66YCULYPYNUEE5CX2QWW7RBNQQJM7WJTPMGA, seg_pos={0 0}, upload=503327]: Done uploading piece. err: <nil>
Dec 29 13:36:48.779911556 (574h7m48.650994591s) [noise=true , address=145.79.227.39:4013   , piece_id=WZJBAXRNGCYMFI672KOD47CNYL6R2DHMTIRMSBH7ZTM6IAY3JWNQ, seg_pos={0 0}, upload=503327]: Done uploading piece. err: <nil>
Dec 29 13:36:48.780554480 (574h7m48.651637516s) [noise=true , address=145.79.227.45:4230   , piece_id=KBYGONO2HPRPZ6WJEC2HRIU3AHPTNDADIYAHKV7G3XQV5QDV4EGA, seg_pos={0 0}, upload=503327]: Done uploading piece. err: <nil>
Dec 29 13:36:48.791401244 (574h7m48.662484279s) [noise=true , address=145.79.227.50:4373   , piece_id=RBFAMNUMC3IEBTRIXGXDZAFSNSIV4E65NESEST7QBBLJKW46HC3Q, seg_pos={0 0}, upload=503327]: Done uploading piece. err: <nil>
Dec 29 13:36:48.792132997 (574h7m48.663216032s) [noise=true , address=145.79.227.15:4876   , piece_id=U7N7JRFGUXWGLKBLGGZHIH3KLMLC75QV5QS3FRPVKZLH423BSW2A, seg_pos={0 0}, upload=503327]: Done uploading piece. err: <nil>
Dec 29 13:36:48.793444725 (574h7m48.664527761s) [noise=true , address=145.79.227.37:4567   , piece_id=LVZXVKKTUFQMSPXVLJU5I5KG6U7DQ457JQP2JSZZE63SIOF6YZ5A, seg_pos={0 0}, upload=503327]: Done uploading piece. err: <nil>
Dec 29 13:36:48.798801485 (574h7m48.66988452s) [noise=true , address=145.79.227.36:4519   , piece_id=Q45YJMQ7FILAA76VQS3OKP5LWK5B6HORI2KZY4OVHEBTETRA64WQ, seg_pos={0 0}, upload=503327]: Done uploading piece. err: <nil>
Dec 29 13:36:48.801158694 (574h7m48.672241729s) [noise=true , address=145.79.227.20:4437   , piece_id=ICZXCCIVIGBUVE24OJDNBDWGAMML2F7S27DCMVENLWV5W5CTXAWA, seg_pos={0 0}, upload=503327]: Done uploading piece. err: <nil>
Dec 29 13:36:48.805316270 (574h7m48.676399305s) [noise=true , address=145.79.227.4:4187    , piece_id=XIY2EU3CFR2PHQTD3JTQ6G56W42RGBRRKG2GCBJTHGLPZUH53KPQ, seg_pos={0 0}, upload=503327]: Done uploading piece. err: <nil>
Dec 29 13:36:49.019378376 (574h7m48.890461411s) [noise=true , address=45.91.116.4:4937     , piece_id=HC3UMDTD3Y532PVH4FAZWCCA4YJESCXFRYDLQ6HNNVJO34K34TPA, seg_pos={0 0}, upload=503327]: Done uploading piece. err: <nil>
Dec 29 13:36:49.032373390 (574h7m48.903456425s) [noise=true , address=77.90.31.19:4842     , piece_id=2UFBA3FKX3LMLYWVCBUN22EYHPFDWQXB4MAYLBDE23BWUHR7LKHQ, seg_pos={0 0}, upload=503327]: Done uploading piece. err: <nil>
Dec 29 13:36:49.038022427 (574h7m48.909105472s) [noise=true , address=45.91.116.11:4118    , piece_id=2W6VDDJ5VEW3VYB32BBCXGEACK3PEVFAUK7QMBW7YX5C3CK2V3MA, seg_pos={0 0}, upload=503327]: Done uploading piece. err: <nil>
Dec 29 13:36:49.038045170 (574h7m48.909128205s) [seg_pos={0 0}, upload=503327]: Segment reached optimal threshold of 13 pieces.
Dec 29 13:36:49.038320124 (574h7m48.909403159s) [noise=true , address=45.91.116.8:4046     , piece_id=PD77P7625XN6PWI32PMIYIAIWZWA6DEJW5YS5PDUKMOELQHGFYPA, seg_pos={0 0}, upload=503327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF
Dec 29 13:36:49.038329281 (574h7m48.909412316s) [noise=true , address=45.91.116.8:4046     , piece_id=PD77P7625XN6PWI32PMIYIAIWZWA6DEJW5YS5PDUKMOELQHGFYPA, seg_pos={0 0}, upload=503327]: Long tail cancellation due to optimal threshold
Dec 29 13:36:49.038429993 (574h7m48.909513038s) [noise=true , address=77.90.31.14:4663     , piece_id=GTW3LDMT3QKPTSAPSJFV2MKU2XN2DFSIS77ZWZCYOLIM5KVBTNYQ, seg_pos={0 0}, upload=503327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF
Dec 29 13:36:49.038448347 (574h7m48.909531403s) [noise=true , address=77.90.31.14:4663     , piece_id=GTW3LDMT3QKPTSAPSJFV2MKU2XN2DFSIS77ZWZCYOLIM5KVBTNYQ, seg_pos={0 0}, upload=503327]: Long tail cancellation due to optimal threshold
Dec 29 13:36:49.038455962 (574h7m48.909539007s) [noise=true , address=45.91.116.5:4299     , piece_id=NFDWU3WN2QCPTV5TZSSMETZMYZ2HTRWKD7I7SDJR7JNSD7FDVKNQ, seg_pos={0 0}, upload=503327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: write tcp 188.166.225.170:56010->45.91.116.5:4299: use of closed network connection; write tcp 188.166.225.170:56010->45.91.116.5:4299: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56010->45.91.116.5:4299: use of closed network connection
Dec 29 13:36:49.038462315 (574h7m48.90954536s) [noise=true , address=45.91.116.5:4299     , piece_id=NFDWU3WN2QCPTV5TZSSMETZMYZ2HTRWKD7I7SDJR7JNSD7FDVKNQ, seg_pos={0 0}, upload=503327]: Long tail cancellation due to optimal threshold
Dec 29 13:36:49.038486099 (574h7m48.909569134s) [seg_pos={0 0}, upload=503327]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 13:36:49.038560000 (574h7m48.909643035s) [upload=503327]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 13:36:49.038564710 (574h7m48.909647745s) [upload=503327]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 13:36:49.323959411 (574h7m49.195042456s) [upload=503327]: Done uploading segments...
Dec 29 13:36:49.323983698 (574h7m49.195066733s) [upload=503327]: Upload finished. err: <nil>
Duration: 5.726732249s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
4 out of 60 is failed (threshold=3) [..X.....X.....................XX............................]
duration: 
47.156898579s
trace: 
https://jaeger.dev.storj.io/trace/77d26141ea11d228
FAIL (2025-12-29 13:37:30 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12J2weU7Bmsr3RdQQ9QMuadT96XReBkyxTeaLJ2SEqVGGFX1PWu, Piece ID: JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ) context deadline exceeded; piecestore: (Node ID: 125sm6iHV94mJFLWrKeyRMH4b1qSMo32nDuW8uQr71xUL7Cema2, Piece ID: INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA) context deadline exceeded; piecestore: (Node ID: 12i57FF97oEddYLZpNxSq5or9WLEquL8ZW1cVUuW4AThBbDqcCL, Piece ID: Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA) context deadline exceeded
======LOGS======
Q2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.692912334 (573h43m3.563995379s) [noise=true , address=77.90.31.16:4759     , piece_id=ARC4HRP57U5VQXZTT63QYLXBPZURI5VMJBIQZOFKGLJBQ3K3KGLQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.693322595 (573h43m3.56440564s) [noise=true , address=77.90.31.15:4711     , piece_id=JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.693739649 (573h43m3.564822694s) [noise=true , address=45.91.116.6:4981     , piece_id=INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.694315887 (573h43m3.565398942s) [noise=true , address=145.79.227.4:4220    , piece_id=SHHJ7EW7DTNOZXVBP6A64VIX7P6AQN62TBCUT3KTH4MIB6UXY3BA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.694765222 (573h43m3.565848267s) [noise=true , address=145.79.227.29:4952   , piece_id=DKYDB2R4DNCGH6T3SLRR2MOZZUPVZZVRL2BKGYOBYXJJVPSZEJ4Q, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.695202545 (573h43m3.56628559s) [noise=true , address=145.79.227.40:4058   , piece_id=KRHH2QZKSDPWTSSKYDVDWGKFJD3LPGP6U7KI2PPJ5T3TXYIUGJWA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.695687528 (573h43m3.566770584s) [noise=true , address=77.90.31.19:4857     , piece_id=Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.696135221 (573h43m3.567218276s) [noise=true , address=45.91.116.8:4051     , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.696703733 (573h43m3.567786778s) [noise=true , address=64.49.13.170:4720    , piece_id=HF6X3OT2EPPDE3DWAPWABVZUR3ERDBWXHG3NGNKEWZAD7G5AN4TA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.698418298 (573h43m3.569501343s) [noise=true , address=145.79.227.36:4520   , piece_id=MYJSXUH563JFUW5Q7YUIWL7466G6ASY5WXR3T2PGUFFCTRTOBYJA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.700198829 (573h43m3.571281874s) [noise=true , address=77.90.31.18:4821     , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.701993145 (573h43m3.57307619s) [noise=true , address=193.149.164.143:4699 , piece_id=UDFDCYO25MOJBTMDE6F2WILXE3HLOB75UJZ4LW7ZMNSCOBJI6UJA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.703884687 (573h43m3.574967732s) [noise=true , address=145.79.227.33:4893   , piece_id=XGLX3DFNAY4F4GPGCGZ4ONDJ7NK62SAKX3PBRWHWI74EF3ZFMUSA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.705687109 (573h43m3.576770154s) [noise=true , address=45.91.116.10:4094    , piece_id=6UUIZNTESBBWJMDQ5ZTCHU7S6YKM4GDKO4ZAC6T2FUTPVGFFDS3A, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.707557651 (573h43m3.578640696s) [noise=true , address=145.79.227.43:4164   , piece_id=J6Q4ZDOIIIBA2FXGZZTD6AYS7KGAWSRXXUBAPDPQWYKNU5GNXBBQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:05.168247719 (573h43m5.039330754s) [upload=502975]: Next returned nil segment
Dec 29 13:12:05.168259962 (573h43m5.039343007s) [upload=502975]: Waiting for error group managing segments...
Dec 29 13:12:05.558873127 (573h43m5.429956162s) [noise=true , address=145.79.227.4:4220    , piece_id=SHHJ7EW7DTNOZXVBP6A64VIX7P6AQN62TBCUT3KTH4MIB6UXY3BA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.563024312 (573h43m5.434107347s) [noise=true , address=64.49.13.170:4720    , piece_id=HF6X3OT2EPPDE3DWAPWABVZUR3ERDBWXHG3NGNKEWZAD7G5AN4TA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.564115719 (573h43m5.435198755s) [noise=true , address=145.79.227.36:4520   , piece_id=MYJSXUH563JFUW5Q7YUIWL7466G6ASY5WXR3T2PGUFFCTRTOBYJA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.593873402 (573h43m5.464956447s) [noise=true , address=145.79.227.29:4952   , piece_id=DKYDB2R4DNCGH6T3SLRR2MOZZUPVZZVRL2BKGYOBYXJJVPSZEJ4Q, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.898194604 (573h43m5.769277639s) [noise=true , address=45.91.116.10:4094    , piece_id=6UUIZNTESBBWJMDQ5ZTCHU7S6YKM4GDKO4ZAC6T2FUTPVGFFDS3A, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.912845292 (573h43m5.783928337s) [noise=true , address=77.90.31.19:4857     , piece_id=Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.426532207 (573h43m6.297615252s) [noise=true , address=193.149.164.143:4699 , piece_id=UDFDCYO25MOJBTMDE6F2WILXE3HLOB75UJZ4LW7ZMNSCOBJI6UJA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.449589562 (573h43m6.320672607s) [noise=true , address=77.90.31.15:4711     , piece_id=JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.453186841 (573h43m6.324269876s) [noise=true , address=145.79.227.40:4058   , piece_id=KRHH2QZKSDPWTSSKYDVDWGKFJD3LPGP6U7KI2PPJ5T3TXYIUGJWA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.510769276 (573h43m6.381852321s) [noise=true , address=145.79.227.43:4164   , piece_id=J6Q4ZDOIIIBA2FXGZZTD6AYS7KGAWSRXXUBAPDPQWYKNU5GNXBBQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.518868879 (573h43m6.389951924s) [noise=true , address=145.79.227.33:4893   , piece_id=XGLX3DFNAY4F4GPGCGZ4ONDJ7NK62SAKX3PBRWHWI74EF3ZFMUSA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.519238043 (573h43m6.390321098s) [noise=true , address=45.91.116.6:4981     , piece_id=INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.523211328 (573h43m6.394294373s) [noise=true , address=77.90.31.16:4759     , piece_id=ARC4HRP57U5VQXZTT63QYLXBPZURI5VMJBIQZOFKGLJBQ3K3KGLQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.767869783 (573h43m6.638952818s) [noise=true , address=145.79.227.34:4924   , piece_id=MAQXQTAGXQ7AEPUCESUSUGMHSDMMXS7HHKAEA7B3HW5X66PDUF5A, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.767889431 (573h43m6.638972476s) [seg_pos={0 0}, upload=502975]: Segment reached optimal threshold of 14 pieces.
Dec 29 13:12:06.768102116 (573h43m6.639185161s) [noise=true , address=45.91.116.8:4051     , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1preT33ei4yVXXYVLHhaUbHbiyL14oHFEiof6cCWPig83W6CT3): protocol: expected piece hash; context canceled; EOF
Dec 29 13:12:06.768108257 (573h43m6.639191302s) [noise=true , address=45.91.116.8:4051     , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768219208 (573h43m6.639302254s) [noise=true , address=77.90.31.18:4821     , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH): protocol: expected piece hash; context canceled; EOF
Dec 29 13:12:06.768224789 (573h43m6.639307824s) [noise=true , address=77.90.31.18:4821     , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768384213 (573h43m6.639467258s) [noise=true , address=64.49.13.169:4711    , piece_id=4F4T4YKXMT7NF2C5KBUGJHQJGZL4VCKNKYB266PCVI3UACHJXZ5Q, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection
Dec 29 13:12:06.768399261 (573h43m6.639482296s) [noise=true , address=64.49.13.169:4711    , piece_id=4F4T4YKXMT7NF2C5KBUGJHQJGZL4VCKNKYB266PCVI3UACHJXZ5Q, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768732636 (573h43m6.639815671s) [noise=true , address=64.49.13.169:4709    , piece_id=TBHHX5ES2A6NQRGP25HIC7J7IBWLOJGQ2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:35310->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:35310->64.49.13.169:4709: use of closed network connection
Dec 29 13:12:06.768745460 (573h43m6.639828505s) [noise=true , address=64.49.13.169:4709    , piece_id=TBHHX5ES2A6NQRGP25HIC7J7IBWLOJGQ2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768784586 (573h43m6.639867631s) [seg_pos={0 0}, upload=502975]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 13:12:06.768880217 (573h43m6.639963252s) [upload=502975]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 13:12:06.768884906 (573h43m6.639967941s) [upload=502975]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 13:12:07.067188330 (573h43m6.938271365s) [upload=502975]: Done uploading segments...
Dec 29 13:12:07.067251099 (573h43m6.938334134s) [upload=502975]: Upload finished. err: <nil>
Duration: 47.156922926s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
5 out of 60 is failed (threshold=3) [.........X....X.X....X.X....................................]
duration: 
20.921461119s
trace: 
https://jaeger.dev.storj.io/trace/312ddb07b2dee82a
FAIL (2025-12-29 13:37:30 UTC) 
orbiter warning: context deadline exceeded
Duration: 20.92148707s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
31 out of 60 is failed (threshold=3) [.X.X.X.XXX.X.X.X.XX..X.XXX.XX..X.XX..XXX...X..X...X...X.XXXX]
duration: 
30.003816988s
trace: 
https://jaeger.dev.storj.io/trace/627523b37459ea01
FAIL (2025-12-29 13:37:15 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 29 13:36:46.538550545 (574h7m46.40963358s) [upload=503327]: Starting upload
Dec 29 13:36:46.538649083 (574h7m46.409732118s) [upload=503327]: Done starting upload
Dec 29 13:36:46.538691995 (574h7m46.40977504s) [upload=503327]: Uploading segments...
Dec 29 13:36:46.538933555 (574h7m46.41001659s) [upload=503327]: Got next segment. Inline: false
Dec 29 13:36:46.539704553 (574h7m46.410787588s) [upload=503327]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 13:36:46.539712998 (574h7m46.410796044s) [upload=503327]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 13:36:46.784755246 (574h7m46.655838281s) [seg_pos={0 0}, upload=503327]: Begin upload segment...
Dec 29 13:36:46.784794711 (574h7m46.655877746s) [seg_pos={0 0}, upload=503327]: Done begin upload segment.
Dec 29 13:36:46.784825379 (574h7m46.655908415s) [noise=true , address=45.91.116.5:4299     , piece_id=NFDWU3WN2QCPTV5TZSSMETZMYZ2HTRWKD7I7SDJR7JNSD7FDVKNQ, seg_pos={0 0}, upload=503327]: Uploading piece...
Dec 29 13:36:46.784943334 (574h7m46.656026379s) [noise=true , address=145.79.227.4:4187    , piece_id=XIY2EU3CFR2PHQTD3JTQ6G56W42RGBRRKG2GCBJTHGLPZUH53KPQ, seg_pos={0 0}, upload=503327]: Uploading piece...
Dec 29 13:36:46.785482059 (574h7m46.656565124s) [noise=true , address=145.79.227.36:4519   , piece_id=Q45YJMQ7FILAA76VQS3OKP5LWK5B6HORI2KZY4OVHEBTETRA64WQ, seg_pos={0 0}, upload=503327]: Uploading piece...
Dec 29 13:36:46.785945172 (574h7m46.657028207s) [noise=true , address=77.90.31.19:4842     , piece_id=2UFBA3FKX3LMLYWVCBUN22EYHPFDWQXB4MAYLBDE23BWUHR7LKHQ, seg_pos={0 0}, upload=503327]: Uploading piece...
Dec 29 13:36:46.786352827 (574h7m46.657435862s) [noise=true , address=145.79.227.15:4876   , piece_id=U7N7JRFGUXWGLKBLGGZHIH3KLMLC75QV5QS3FRPVKZLH423BSW2A, seg_pos={0 0}, upload=503327]: Uploading piece...
Dec 29 13:36:46.786638151 (574h7m46.657721186s) [noise=true , address=45.91.116.8:4046     , piece_id=PD77P7625XN6PWI32PMIYIAIWZWA6DEJW5YS5PDUKMOELQHGFYPA, seg_pos={0 0}, upload=503327]: Uploading piece...
Dec 29 13:36:46.787020309 (574h7m46.658103344s) [noise=true , address=145.79.227.50:4373   , piece_id=RBFAMNUMC3IEBTRIXGXDZAFSNSIV4E65NESEST7QBBLJKW46HC3Q, seg_pos={0 0}, upload=503327]: Uploading piece...
Dec 29 13:36:46.787106022 (574h7m46.658189057s) [noise=true , address=77.90.31.14:4663     , piece_id=GTW3LDMT3QKPTSAPSJFV2MKU2XN2DFSIS77ZWZCYOLIM5KVBTNYQ, seg_pos={0 0}, upload=503327]: Uploading piece...
Dec 29 13:36:46.787643516 (574h7m46.658726551s) [noise=true , address=145.79.227.37:4567   , piece_id=LVZXVKKTUFQMSPXVLJU5I5KG6U7DQ457JQP2JSZZE63SIOF6YZ5A, seg_pos={0 0}, upload=503327]: Uploading piece...
Dec 29 13:36:46.787722436 (574h7m46.658805471s) [noise=true , address=145.79.227.39:4013   , piece_id=WZJBAXRNGCYMFI672KOD47CNYL6R2DHMTIRMSBH7ZTM6IAY3JWNQ, seg_pos={0 0}, upload=503327]: Uploading piece...
Dec 29 13:36:46.788257053 (574h7m46.659340088s) [noise=true , address=145.79.227.20:4437   , piece_id=ICZXCCIVIGBUVE24OJDNBDWGAMML2F7S27DCMVENLWV5W5CTXAWA, seg_pos={0 0}, upload=503327]: Uploading piece...
Dec 29 13:36:46.788285327 (574h7m46.659368362s) [noise=true , address=45.91.116.4:4937     , piece_id=HC3UMDTD3Y532PVH4FAZWCCA4YJESCXFRYDLQ6HNNVJO34K34TPA, seg_pos={0 0}, upload=503327]: Uploading piece...
Dec 29 13:36:46.788882504 (574h7m46.659965549s) [noise=true , address=145.79.227.13:4802   , piece_id=YHXPFWFP6GOLTFYW66YCULYPYNUEE5CX2QWW7RBNQQJM7WJTPMGA, seg_pos={0 0}, upload=503327]: Uploading piece...
Dec 29 13:36:46.790211465 (574h7m46.66129451s) [noise=true , address=145.79.227.45:4230   , piece_id=KBYGONO2HPRPZ6WJEC2HRIU3AHPTNDADIYAHKV7G3XQV5QDV4EGA, seg_pos={0 0}, upload=503327]: Uploading piece...
Dec 29 13:36:46.790644229 (574h7m46.661727275s) [noise=true , address=145.79.227.3:4149    , piece_id=KYLZRXN33U5EYWLCTG6IBBTUFMWCT3Y3YCLE65Q7XHSENKJCSZCA, seg_pos={0 0}, upload=503327]: Uploading piece...
Dec 29 13:36:46.791079989 (574h7m46.662163034s) [noise=true , address=45.91.116.11:4118    , piece_id=2W6VDDJ5VEW3VYB32BBCXGEACK3PEVFAUK7QMBW7YX5C3CK2V3MA, seg_pos={0 0}, upload=503327]: Uploading piece...
Dec 29 13:36:48.486010585 (574h7m48.35709362s) [upload=503327]: Next returned nil segment
Dec 29 13:36:48.486028931 (574h7m48.357111976s) [upload=503327]: Waiting for error group managing segments...
Dec 29 13:36:48.768858048 (574h7m48.639941084s) [noise=true , address=145.79.227.3:4149    , piece_id=KYLZRXN33U5EYWLCTG6IBBTUFMWCT3Y3YCLE65Q7XHSENKJCSZCA, seg_pos={0 0}, upload=503327]: Done uploading piece. err: <nil>
Dec 29 13:36:48.770501598 (574h7m48.641584633s) [noise=true , address=145.79.227.13:4802   , piece_id=YHXPFWFP6GOLTFYW66YCULYPYNUEE5CX2QWW7RBNQQJM7WJTPMGA, seg_pos={0 0}, upload=503327]: Done uploading piece. err: <nil>
Dec 29 13:36:48.779911556 (574h7m48.650994591s) [noise=true , address=145.79.227.39:4013   , piece_id=WZJBAXRNGCYMFI672KOD47CNYL6R2DHMTIRMSBH7ZTM6IAY3JWNQ, seg_pos={0 0}, upload=503327]: Done uploading piece. err: <nil>
Dec 29 13:36:48.780554480 (574h7m48.651637516s) [noise=true , address=145.79.227.45:4230   , piece_id=KBYGONO2HPRPZ6WJEC2HRIU3AHPTNDADIYAHKV7G3XQV5QDV4EGA, seg_pos={0 0}, upload=503327]: Done uploading piece. err: <nil>
Dec 29 13:36:48.791401244 (574h7m48.662484279s) [noise=true , address=145.79.227.50:4373   , piece_id=RBFAMNUMC3IEBTRIXGXDZAFSNSIV4E65NESEST7QBBLJKW46HC3Q, seg_pos={0 0}, upload=503327]: Done uploading piece. err: <nil>
Dec 29 13:36:48.792132997 (574h7m48.663216032s) [noise=true , address=145.79.227.15:4876   , piece_id=U7N7JRFGUXWGLKBLGGZHIH3KLMLC75QV5QS3FRPVKZLH423BSW2A, seg_pos={0 0}, upload=503327]: Done uploading piece. err: <nil>
Dec 29 13:36:48.793444725 (574h7m48.664527761s) [noise=true , address=145.79.227.37:4567   , piece_id=LVZXVKKTUFQMSPXVLJU5I5KG6U7DQ457JQP2JSZZE63SIOF6YZ5A, seg_pos={0 0}, upload=503327]: Done uploading piece. err: <nil>
Dec 29 13:36:48.798801485 (574h7m48.66988452s) [noise=true , address=145.79.227.36:4519   , piece_id=Q45YJMQ7FILAA76VQS3OKP5LWK5B6HORI2KZY4OVHEBTETRA64WQ, seg_pos={0 0}, upload=503327]: Done uploading piece. err: <nil>
Dec 29 13:36:48.801158694 (574h7m48.672241729s) [noise=true , address=145.79.227.20:4437   , piece_id=ICZXCCIVIGBUVE24OJDNBDWGAMML2F7S27DCMVENLWV5W5CTXAWA, seg_pos={0 0}, upload=503327]: Done uploading piece. err: <nil>
Dec 29 13:36:48.805316270 (574h7m48.676399305s) [noise=true , address=145.79.227.4:4187    , piece_id=XIY2EU3CFR2PHQTD3JTQ6G56W42RGBRRKG2GCBJTHGLPZUH53KPQ, seg_pos={0 0}, upload=503327]: Done uploading piece. err: <nil>
Dec 29 13:36:49.019378376 (574h7m48.890461411s) [noise=true , address=45.91.116.4:4937     , piece_id=HC3UMDTD3Y532PVH4FAZWCCA4YJESCXFRYDLQ6HNNVJO34K34TPA, seg_pos={0 0}, upload=503327]: Done uploading piece. err: <nil>
Dec 29 13:36:49.032373390 (574h7m48.903456425s) [noise=true , address=77.90.31.19:4842     , piece_id=2UFBA3FKX3LMLYWVCBUN22EYHPFDWQXB4MAYLBDE23BWUHR7LKHQ, seg_pos={0 0}, upload=503327]: Done uploading piece. err: <nil>
Dec 29 13:36:49.038022427 (574h7m48.909105472s) [noise=true , address=45.91.116.11:4118    , piece_id=2W6VDDJ5VEW3VYB32BBCXGEACK3PEVFAUK7QMBW7YX5C3CK2V3MA, seg_pos={0 0}, upload=503327]: Done uploading piece. err: <nil>
Dec 29 13:36:49.038045170 (574h7m48.909128205s) [seg_pos={0 0}, upload=503327]: Segment reached optimal threshold of 13 pieces.
Dec 29 13:36:49.038320124 (574h7m48.909403159s) [noise=true , address=45.91.116.8:4046     , piece_id=PD77P7625XN6PWI32PMIYIAIWZWA6DEJW5YS5PDUKMOELQHGFYPA, seg_pos={0 0}, upload=503327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF
Dec 29 13:36:49.038329281 (574h7m48.909412316s) [noise=true , address=45.91.116.8:4046     , piece_id=PD77P7625XN6PWI32PMIYIAIWZWA6DEJW5YS5PDUKMOELQHGFYPA, seg_pos={0 0}, upload=503327]: Long tail cancellation due to optimal threshold
Dec 29 13:36:49.038429993 (574h7m48.909513038s) [noise=true , address=77.90.31.14:4663     , piece_id=GTW3LDMT3QKPTSAPSJFV2MKU2XN2DFSIS77ZWZCYOLIM5KVBTNYQ, seg_pos={0 0}, upload=503327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF
Dec 29 13:36:49.038448347 (574h7m48.909531403s) [noise=true , address=77.90.31.14:4663     , piece_id=GTW3LDMT3QKPTSAPSJFV2MKU2XN2DFSIS77ZWZCYOLIM5KVBTNYQ, seg_pos={0 0}, upload=503327]: Long tail cancellation due to optimal threshold
Dec 29 13:36:49.038455962 (574h7m48.909539007s) [noise=true , address=45.91.116.5:4299     , piece_id=NFDWU3WN2QCPTV5TZSSMETZMYZ2HTRWKD7I7SDJR7JNSD7FDVKNQ, seg_pos={0 0}, upload=503327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: write tcp 188.166.225.170:56010->45.91.116.5:4299: use of closed network connection; write tcp 188.166.225.170:56010->45.91.116.5:4299: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56010->45.91.116.5:4299: use of closed network connection
Dec 29 13:36:49.038462315 (574h7m48.90954536s) [noise=true , address=45.91.116.5:4299     , piece_id=NFDWU3WN2QCPTV5TZSSMETZMYZ2HTRWKD7I7SDJR7JNSD7FDVKNQ, seg_pos={0 0}, upload=503327]: Long tail cancellation due to optimal threshold
Dec 29 13:36:49.038486099 (574h7m48.909569134s) [seg_pos={0 0}, upload=503327]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 13:36:49.038560000 (574h7m48.909643035s) [upload=503327]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 13:36:49.038564710 (574h7m48.909647745s) [upload=503327]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 13:36:49.323959411 (574h7m49.195042456s) [upload=503327]: Done uploading segments...
Dec 29 13:36:49.323983698 (574h7m49.195066733s) [upload=503327]: Upload finished. err: <nil>
Duration: 30.003838608s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
53 out of 60 is failed (threshold=6) [XX.XXX.XX.XXXXXXX.XX.XXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXX.XXX]
duration: 
5.00090628s
trace: 
https://jaeger.dev.storj.io/trace/23dc4b5645e396f9
FAIL (2025-12-29 13:36:58 UTC) 
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000931559s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
5 out of 60 is failed (threshold=3) [..........X....X.X....X.X...................................]
duration: 
16.706770307s
trace: 
https://jaeger.dev.storj.io/trace/312ddb07b2dee82a
FAIL (2025-12-29 13:36:43 UTC) 
orbiter warning: context deadline exceeded
Duration: 16.706795907s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
52 out of 60 is failed (threshold=6) [.XX.XXX.XX.XXXXXXX.XX.XXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXX.XX]
duration: 
5.00126678s
trace: 
https://jaeger.dev.storj.io/trace/58c6bf587060dff6
FAIL (2025-12-29 13:36:11 UTC) 
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001291826s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
5 out of 60 is failed (threshold=3) [...........X....X.X....X.X..................................]
duration: 
25.122982726s
trace: 
https://jaeger.dev.storj.io/trace/312ddb07b2dee82a
FAIL (2025-12-29 13:36:10 UTC) 
orbiter warning: context deadline exceeded
Duration: 25.123009576s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
30 out of 60 is failed (threshold=3) [..X.X.X.XXX.X.X.X.XX..X.XXX.XX..X.XX..XXX...X..X...X...X.XXX]
duration: 
30.003478125s
trace: 
https://jaeger.dev.storj.io/trace/9ce8513e3133d42
FAIL (2025-12-29 13:36:08 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: AHNMESHSPMWXY2STQ2X3PWT2LS4HSFAOE5Y2QMYOCJDHJP7VOFUA) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: XY6YX54CSBKZ4EQUDR322ZY26IQV3BAJTLVAFRRFYISHH7K7MIOQ) context deadline exceeded
======LOGS======
Dec 29 13:35:39.644829523 (574h6m39.515912559s) [upload=503314]: Starting upload
Dec 29 13:35:39.644910167 (574h6m39.515993212s) [upload=503314]: Done starting upload
Dec 29 13:35:39.644979268 (574h6m39.516062303s) [upload=503314]: Uploading segments...
Dec 29 13:35:39.645175432 (574h6m39.516258477s) [upload=503314]: Got next segment. Inline: false
Dec 29 13:35:39.645237821 (574h6m39.516320866s) [upload=503314]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 13:35:39.645243933 (574h6m39.516326978s) [upload=503314]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 13:35:39.888381872 (574h6m39.759464907s) [seg_pos={0 0}, upload=503314]: Begin upload segment...
Dec 29 13:35:39.888434913 (574h6m39.759517948s) [seg_pos={0 0}, upload=503314]: Done begin upload segment.
Dec 29 13:35:39.888520245 (574h6m39.75960328s) [noise=true , address=145.79.227.33:4870   , piece_id=S3JDJZTN44UJWTR5EOTAV4EDQ34LSXPTB43XXVR37TJDEB5OC3NA, seg_pos={0 0}, upload=503314]: Uploading piece...
Dec 29 13:35:39.888561093 (574h6m39.759644138s) [noise=true , address=45.91.116.9:4263     , piece_id=XY6YX54CSBKZ4EQUDR322ZY26IQV3BAJTLVAFRRFYISHH7K7MIOQ, seg_pos={0 0}, upload=503314]: Uploading piece...
Dec 29 13:35:39.889156597 (574h6m39.760239632s) [noise=true , address=145.79.227.41:4087   , piece_id=YBFN6522QODUFPHBA74YYNBHQ3FYJ3D6KZIVUIAAWCJU4GON7SPA, seg_pos={0 0}, upload=503314]: Uploading piece...
Dec 29 13:35:39.889217383 (574h6m39.760300418s) [noise=true , address=145.79.227.39:4015   , piece_id=ANFOTB7TQJSMFWVQOTI4KGTICKBV5FKFRNYSWY6XKQOS4U233CRQ, seg_pos={0 0}, upload=503314]: Uploading piece...
Dec 29 13:35:39.889550247 (574h6m39.760633312s) [noise=true , address=145.79.227.13:4803   , piece_id=V6AUMK4YGLM72556G7J4JROTBRKYF7NKBLYG7ZIYSGVG3Z3GCPDQ, seg_pos={0 0}, upload=503314]: Uploading piece...
Dec 29 13:35:39.889999362 (574h6m39.761082628s) [noise=true , address=145.79.227.3:4151    , piece_id=YHL63SYIUXSPJSFIQW5UHTK3OSWZ5TMS54IIYSRCXZKZEDHMSQMQ, seg_pos={0 0}, upload=503314]: Uploading piece...
Dec 29 13:35:39.890310634 (574h6m39.761393669s) [noise=true , address=45.91.116.10:4084    , piece_id=AHNMESHSPMWXY2STQ2X3PWT2LS4HSFAOE5Y2QMYOCJDHJP7VOFUA, seg_pos={0 0}, upload=503314]: Uploading piece...
Dec 29 13:35:39.890496779 (574h6m39.761579814s) [noise=true , address=45.91.116.13:4153    , piece_id=XPOCOVOZ3XUNDNDVMT2GDP6RFNDUACYPXF7XIBUCTUKUN6HIETIQ, seg_pos={0 0}, upload=503314]: Uploading piece...
Dec 29 13:35:39.890690047 (574h6m39.761773092s) [noise=true , address=145.79.227.17:4970   , piece_id=5GCEAB4RIIQJPO26KOELNZIMJEUBZPR2XYKGNNTKWNXCQ7DK3NRQ, seg_pos={0 0}, upload=503314]: Uploading piece...
Dec 29 13:35:39.890904766 (574h6m39.761987801s) [noise=true , address=145.79.227.46:4603   , piece_id=ERNKRETRBRX3AEEKFCZR4ICDDJ3UTSIYX5BN6EKGSNQS2QCPVUTA, seg_pos={0 0}, upload=503314]: Uploading piece...
Dec 29 13:35:39.891043511 (574h6m39.762126646s) [noise=true , address=45.91.116.5:4299     , piece_id=7KDZVON64IWGQK4IGVMV5EQNMVJZEC4H4IRGKRC47MXFKYXNCG6Q, seg_pos={0 0}, upload=503314]: Uploading piece...
Dec 29 13:35:39.891365063 (574h6m39.762448098s) [noise=true , address=45.91.116.14:4191    , piece_id=UWPI557VSZRCX4TDCCBICIJVUNUCFWPH6ZYNJNKW6FKY5EMHV6FQ, seg_pos={0 0}, upload=503314]: Uploading piece...
Dec 29 13:35:39.891389589 (574h6m39.762472624s) [noise=true , address=145.79.227.22:4511   , piece_id=DREBMFACEID3KVEXLFNY5MDYXSVTSV4AEYFQ64UC7IXH3BR63JRQ, seg_pos={0 0}, upload=503314]: Uploading piece...
Dec 29 13:35:39.891706813 (574h6m39.762789849s) [noise=true , address=145.79.227.2:4113    , piece_id=7CBFZKDDDVZEMQP4BMTTR7PXRXYK2LWR52TF5HBYJJ3B4GWLN2TQ, seg_pos={0 0}, upload=503314]: Uploading piece...
Dec 29 13:35:39.891776396 (574h6m39.762859431s) [noise=true , address=77.90.31.14:4663     , piece_id=MATNMDTK5BPB2RY7Q5ZNJWQHDSVLTQLVSLZLFB3MWKR3Z5NOJQRQ, seg_pos={0 0}, upload=503314]: Uploading piece...
Dec 29 13:35:39.892050237 (574h6m39.763133272s) [noise=true , address=145.79.227.35:4483   , piece_id=OEXOCK7V4HQX6XY2GDLFVE3TDQMJZ53QX4OII7AKIJEOF7627H6A, seg_pos={0 0}, upload=503314]: Uploading piece...
Dec 29 13:35:41.499901843 (574h6m41.370984898s) [upload=503314]: Next returned nil segment
Dec 29 13:35:41.499919506 (574h6m41.371002541s) [upload=503314]: Waiting for error group managing segments...
Dec 29 13:35:41.877072161 (574h6m41.748155206s) [noise=true , address=145.79.227.2:4113    , piece_id=7CBFZKDDDVZEMQP4BMTTR7PXRXYK2LWR52TF5HBYJJ3B4GWLN2TQ, seg_pos={0 0}, upload=503314]: Done uploading piece. err: <nil>
Dec 29 13:35:41.878490542 (574h6m41.749573587s) [noise=true , address=145.79.227.3:4151    , piece_id=YHL63SYIUXSPJSFIQW5UHTK3OSWZ5TMS54IIYSRCXZKZEDHMSQMQ, seg_pos={0 0}, upload=503314]: Done uploading piece. err: <nil>
Dec 29 13:35:41.879694334 (574h6m41.750777369s) [noise=true , address=145.79.227.17:4970   , piece_id=5GCEAB4RIIQJPO26KOELNZIMJEUBZPR2XYKGNNTKWNXCQ7DK3NRQ, seg_pos={0 0}, upload=503314]: Done uploading piece. err: <nil>
Dec 29 13:35:41.883045596 (574h6m41.754128641s) [noise=true , address=145.79.227.41:4087   , piece_id=YBFN6522QODUFPHBA74YYNBHQ3FYJ3D6KZIVUIAAWCJU4GON7SPA, seg_pos={0 0}, upload=503314]: Done uploading piece. err: <nil>
Dec 29 13:35:41.889503703 (574h6m41.760586748s) [noise=true , address=145.79.227.46:4603   , piece_id=ERNKRETRBRX3AEEKFCZR4ICDDJ3UTSIYX5BN6EKGSNQS2QCPVUTA, seg_pos={0 0}, upload=503314]: Done uploading piece. err: <nil>
Dec 29 13:35:41.894976493 (574h6m41.766059528s) [noise=true , address=145.79.227.22:4511   , piece_id=DREBMFACEID3KVEXLFNY5MDYXSVTSV4AEYFQ64UC7IXH3BR63JRQ, seg_pos={0 0}, upload=503314]: Done uploading piece. err: <nil>
Dec 29 13:35:41.897200418 (574h6m41.768283463s) [noise=true , address=145.79.227.13:4803   , piece_id=V6AUMK4YGLM72556G7J4JROTBRKYF7NKBLYG7ZIYSGVG3Z3GCPDQ, seg_pos={0 0}, upload=503314]: Done uploading piece. err: <nil>
Dec 29 13:35:41.898729280 (574h6m41.769812325s) [noise=true , address=145.79.227.39:4015   , piece_id=ANFOTB7TQJSMFWVQOTI4KGTICKBV5FKFRNYSWY6XKQOS4U233CRQ, seg_pos={0 0}, upload=503314]: Done uploading piece. err: <nil>
Dec 29 13:35:42.156421153 (574h6m42.027504208s) [noise=true , address=45.91.116.10:4084    , piece_id=AHNMESHSPMWXY2STQ2X3PWT2LS4HSFAOE5Y2QMYOCJDHJP7VOFUA, seg_pos={0 0}, upload=503314]: Done uploading piece. err: <nil>
Dec 29 13:35:42.215582164 (574h6m42.086665199s) [noise=true , address=45.91.116.13:4153    , piece_id=XPOCOVOZ3XUNDNDVMT2GDP6RFNDUACYPXF7XIBUCTUKUN6HIETIQ, seg_pos={0 0}, upload=503314]: Done uploading piece. err: <nil>
Dec 29 13:35:42.231272781 (574h6m42.102355826s) [noise=true , address=45.91.116.5:4299     , piece_id=7KDZVON64IWGQK4IGVMV5EQNMVJZEC4H4IRGKRC47MXFKYXNCG6Q, seg_pos={0 0}, upload=503314]: Done uploading piece. err: <nil>
Dec 29 13:35:42.254176944 (574h6m42.12525998s) [noise=true , address=77.90.31.14:4663     , piece_id=MATNMDTK5BPB2RY7Q5ZNJWQHDSVLTQLVSLZLFB3MWKR3Z5NOJQRQ, seg_pos={0 0}, upload=503314]: Done uploading piece. err: <nil>
Dec 29 13:35:42.279805264 (574h6m42.150888299s) [noise=true , address=45.91.116.9:4263     , piece_id=XY6YX54CSBKZ4EQUDR322ZY26IQV3BAJTLVAFRRFYISHH7K7MIOQ, seg_pos={0 0}, upload=503314]: Done uploading piece. err: <nil>
Dec 29 13:35:42.279827016 (574h6m42.150910051s) [seg_pos={0 0}, upload=503314]: Segment reached optimal threshold of 13 pieces.
Dec 29 13:35:42.280037737 (574h6m42.151120782s) [noise=true , address=145.79.227.35:4483   , piece_id=OEXOCK7V4HQX6XY2GDLFVE3TDQMJZ53QX4OII7AKIJEOF7627H6A, seg_pos={0 0}, upload=503314]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF
Dec 29 13:35:42.280048227 (574h6m42.151131272s) [noise=true , address=145.79.227.35:4483   , piece_id=OEXOCK7V4HQX6XY2GDLFVE3TDQMJZ53QX4OII7AKIJEOF7627H6A, seg_pos={0 0}, upload=503314]: Long tail cancellation due to optimal threshold
Dec 29 13:35:42.280064537 (574h6m42.151147582s) [noise=true , address=45.91.116.14:4191    , piece_id=UWPI557VSZRCX4TDCCBICIJVUNUCFWPH6ZYNJNKW6FKY5EMHV6FQ, seg_pos={0 0}, upload=503314]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:47118->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:47118->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47118->45.91.116.14:4191: use of closed network connection
Dec 29 13:35:42.280103713 (574h6m42.151186778s) [noise=true , address=45.91.116.14:4191    , piece_id=UWPI557VSZRCX4TDCCBICIJVUNUCFWPH6ZYNJNKW6FKY5EMHV6FQ, seg_pos={0 0}, upload=503314]: Long tail cancellation due to optimal threshold
Dec 29 13:35:42.280186961 (574h6m42.151269996s) [noise=true , address=145.79.227.33:4870   , piece_id=S3JDJZTN44UJWTR5EOTAV4EDQ34LSXPTB43XXVR37TJDEB5OC3NA, seg_pos={0 0}, upload=503314]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF
Dec 29 13:35:42.280195106 (574h6m42.151278141s) [noise=true , address=145.79.227.33:4870   , piece_id=S3JDJZTN44UJWTR5EOTAV4EDQ34LSXPTB43XXVR37TJDEB5OC3NA, seg_pos={0 0}, upload=503314]: Long tail cancellation due to optimal threshold
Dec 29 13:35:42.280216006 (574h6m42.151299041s) [seg_pos={0 0}, upload=503314]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 13:35:42.280292812 (574h6m42.151375847s) [upload=503314]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 13:35:42.280298794 (574h6m42.151381829s) [upload=503314]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 13:35:42.576603381 (574h6m42.447686416s) [upload=503314]: Done uploading segments...
Dec 29 13:35:42.576622057 (574h6m42.447705092s) [upload=503314]: Upload finished. err: <nil>
Duration: 30.003508151s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
4 out of 60 is failed (threshold=3) [...X.....X.....................XX...........................]
duration: 
15.517014434s
trace: 
https://jaeger.dev.storj.io/trace/77d26141ea11d228
FAIL (2025-12-29 13:35:48 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12J2weU7Bmsr3RdQQ9QMuadT96XReBkyxTeaLJ2SEqVGGFX1PWu, Piece ID: JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ) context deadline exceeded; piecestore: (Node ID: 125sm6iHV94mJFLWrKeyRMH4b1qSMo32nDuW8uQr71xUL7Cema2, Piece ID: INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA) context deadline exceeded; piecestore: (Node ID: 12i57FF97oEddYLZpNxSq5or9WLEquL8ZW1cVUuW4AThBbDqcCL, Piece ID: Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA) context deadline exceeded
======LOGS======
Q2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.692912334 (573h43m3.563995379s) [noise=true , address=77.90.31.16:4759     , piece_id=ARC4HRP57U5VQXZTT63QYLXBPZURI5VMJBIQZOFKGLJBQ3K3KGLQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.693322595 (573h43m3.56440564s) [noise=true , address=77.90.31.15:4711     , piece_id=JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.693739649 (573h43m3.564822694s) [noise=true , address=45.91.116.6:4981     , piece_id=INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.694315887 (573h43m3.565398942s) [noise=true , address=145.79.227.4:4220    , piece_id=SHHJ7EW7DTNOZXVBP6A64VIX7P6AQN62TBCUT3KTH4MIB6UXY3BA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.694765222 (573h43m3.565848267s) [noise=true , address=145.79.227.29:4952   , piece_id=DKYDB2R4DNCGH6T3SLRR2MOZZUPVZZVRL2BKGYOBYXJJVPSZEJ4Q, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.695202545 (573h43m3.56628559s) [noise=true , address=145.79.227.40:4058   , piece_id=KRHH2QZKSDPWTSSKYDVDWGKFJD3LPGP6U7KI2PPJ5T3TXYIUGJWA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.695687528 (573h43m3.566770584s) [noise=true , address=77.90.31.19:4857     , piece_id=Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.696135221 (573h43m3.567218276s) [noise=true , address=45.91.116.8:4051     , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.696703733 (573h43m3.567786778s) [noise=true , address=64.49.13.170:4720    , piece_id=HF6X3OT2EPPDE3DWAPWABVZUR3ERDBWXHG3NGNKEWZAD7G5AN4TA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.698418298 (573h43m3.569501343s) [noise=true , address=145.79.227.36:4520   , piece_id=MYJSXUH563JFUW5Q7YUIWL7466G6ASY5WXR3T2PGUFFCTRTOBYJA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.700198829 (573h43m3.571281874s) [noise=true , address=77.90.31.18:4821     , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.701993145 (573h43m3.57307619s) [noise=true , address=193.149.164.143:4699 , piece_id=UDFDCYO25MOJBTMDE6F2WILXE3HLOB75UJZ4LW7ZMNSCOBJI6UJA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.703884687 (573h43m3.574967732s) [noise=true , address=145.79.227.33:4893   , piece_id=XGLX3DFNAY4F4GPGCGZ4ONDJ7NK62SAKX3PBRWHWI74EF3ZFMUSA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.705687109 (573h43m3.576770154s) [noise=true , address=45.91.116.10:4094    , piece_id=6UUIZNTESBBWJMDQ5ZTCHU7S6YKM4GDKO4ZAC6T2FUTPVGFFDS3A, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.707557651 (573h43m3.578640696s) [noise=true , address=145.79.227.43:4164   , piece_id=J6Q4ZDOIIIBA2FXGZZTD6AYS7KGAWSRXXUBAPDPQWYKNU5GNXBBQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:05.168247719 (573h43m5.039330754s) [upload=502975]: Next returned nil segment
Dec 29 13:12:05.168259962 (573h43m5.039343007s) [upload=502975]: Waiting for error group managing segments...
Dec 29 13:12:05.558873127 (573h43m5.429956162s) [noise=true , address=145.79.227.4:4220    , piece_id=SHHJ7EW7DTNOZXVBP6A64VIX7P6AQN62TBCUT3KTH4MIB6UXY3BA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.563024312 (573h43m5.434107347s) [noise=true , address=64.49.13.170:4720    , piece_id=HF6X3OT2EPPDE3DWAPWABVZUR3ERDBWXHG3NGNKEWZAD7G5AN4TA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.564115719 (573h43m5.435198755s) [noise=true , address=145.79.227.36:4520   , piece_id=MYJSXUH563JFUW5Q7YUIWL7466G6ASY5WXR3T2PGUFFCTRTOBYJA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.593873402 (573h43m5.464956447s) [noise=true , address=145.79.227.29:4952   , piece_id=DKYDB2R4DNCGH6T3SLRR2MOZZUPVZZVRL2BKGYOBYXJJVPSZEJ4Q, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.898194604 (573h43m5.769277639s) [noise=true , address=45.91.116.10:4094    , piece_id=6UUIZNTESBBWJMDQ5ZTCHU7S6YKM4GDKO4ZAC6T2FUTPVGFFDS3A, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.912845292 (573h43m5.783928337s) [noise=true , address=77.90.31.19:4857     , piece_id=Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.426532207 (573h43m6.297615252s) [noise=true , address=193.149.164.143:4699 , piece_id=UDFDCYO25MOJBTMDE6F2WILXE3HLOB75UJZ4LW7ZMNSCOBJI6UJA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.449589562 (573h43m6.320672607s) [noise=true , address=77.90.31.15:4711     , piece_id=JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.453186841 (573h43m6.324269876s) [noise=true , address=145.79.227.40:4058   , piece_id=KRHH2QZKSDPWTSSKYDVDWGKFJD3LPGP6U7KI2PPJ5T3TXYIUGJWA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.510769276 (573h43m6.381852321s) [noise=true , address=145.79.227.43:4164   , piece_id=J6Q4ZDOIIIBA2FXGZZTD6AYS7KGAWSRXXUBAPDPQWYKNU5GNXBBQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.518868879 (573h43m6.389951924s) [noise=true , address=145.79.227.33:4893   , piece_id=XGLX3DFNAY4F4GPGCGZ4ONDJ7NK62SAKX3PBRWHWI74EF3ZFMUSA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.519238043 (573h43m6.390321098s) [noise=true , address=45.91.116.6:4981     , piece_id=INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.523211328 (573h43m6.394294373s) [noise=true , address=77.90.31.16:4759     , piece_id=ARC4HRP57U5VQXZTT63QYLXBPZURI5VMJBIQZOFKGLJBQ3K3KGLQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.767869783 (573h43m6.638952818s) [noise=true , address=145.79.227.34:4924   , piece_id=MAQXQTAGXQ7AEPUCESUSUGMHSDMMXS7HHKAEA7B3HW5X66PDUF5A, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.767889431 (573h43m6.638972476s) [seg_pos={0 0}, upload=502975]: Segment reached optimal threshold of 14 pieces.
Dec 29 13:12:06.768102116 (573h43m6.639185161s) [noise=true , address=45.91.116.8:4051     , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1preT33ei4yVXXYVLHhaUbHbiyL14oHFEiof6cCWPig83W6CT3): protocol: expected piece hash; context canceled; EOF
Dec 29 13:12:06.768108257 (573h43m6.639191302s) [noise=true , address=45.91.116.8:4051     , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768219208 (573h43m6.639302254s) [noise=true , address=77.90.31.18:4821     , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH): protocol: expected piece hash; context canceled; EOF
Dec 29 13:12:06.768224789 (573h43m6.639307824s) [noise=true , address=77.90.31.18:4821     , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768384213 (573h43m6.639467258s) [noise=true , address=64.49.13.169:4711    , piece_id=4F4T4YKXMT7NF2C5KBUGJHQJGZL4VCKNKYB266PCVI3UACHJXZ5Q, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection
Dec 29 13:12:06.768399261 (573h43m6.639482296s) [noise=true , address=64.49.13.169:4711    , piece_id=4F4T4YKXMT7NF2C5KBUGJHQJGZL4VCKNKYB266PCVI3UACHJXZ5Q, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768732636 (573h43m6.639815671s) [noise=true , address=64.49.13.169:4709    , piece_id=TBHHX5ES2A6NQRGP25HIC7J7IBWLOJGQ2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:35310->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:35310->64.49.13.169:4709: use of closed network connection
Dec 29 13:12:06.768745460 (573h43m6.639828505s) [noise=true , address=64.49.13.169:4709    , piece_id=TBHHX5ES2A6NQRGP25HIC7J7IBWLOJGQ2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768784586 (573h43m6.639867631s) [seg_pos={0 0}, upload=502975]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 13:12:06.768880217 (573h43m6.639963252s) [upload=502975]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 13:12:06.768884906 (573h43m6.639967941s) [upload=502975]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 13:12:07.067188330 (573h43m6.938271365s) [upload=502975]: Done uploading segments...
Dec 29 13:12:07.067251099 (573h43m6.938334134s) [upload=502975]: Upload finished. err: <nil>
Duration: 15.517028902s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
4 out of 60 is failed (threshold=3) [....X.....X.....................XX..........................]
duration: 
54.90880946s
trace: 
https://jaeger.dev.storj.io/trace/77d26141ea11d228
FAIL (2025-12-29 13:35:32 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12J2weU7Bmsr3RdQQ9QMuadT96XReBkyxTeaLJ2SEqVGGFX1PWu, Piece ID: JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ) context deadline exceeded; piecestore: (Node ID: 125sm6iHV94mJFLWrKeyRMH4b1qSMo32nDuW8uQr71xUL7Cema2, Piece ID: INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA) context deadline exceeded; piecestore: (Node ID: 12i57FF97oEddYLZpNxSq5or9WLEquL8ZW1cVUuW4AThBbDqcCL, Piece ID: Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA) context deadline exceeded
======LOGS======
Q2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.692912334 (573h43m3.563995379s) [noise=true , address=77.90.31.16:4759     , piece_id=ARC4HRP57U5VQXZTT63QYLXBPZURI5VMJBIQZOFKGLJBQ3K3KGLQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.693322595 (573h43m3.56440564s) [noise=true , address=77.90.31.15:4711     , piece_id=JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.693739649 (573h43m3.564822694s) [noise=true , address=45.91.116.6:4981     , piece_id=INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.694315887 (573h43m3.565398942s) [noise=true , address=145.79.227.4:4220    , piece_id=SHHJ7EW7DTNOZXVBP6A64VIX7P6AQN62TBCUT3KTH4MIB6UXY3BA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.694765222 (573h43m3.565848267s) [noise=true , address=145.79.227.29:4952   , piece_id=DKYDB2R4DNCGH6T3SLRR2MOZZUPVZZVRL2BKGYOBYXJJVPSZEJ4Q, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.695202545 (573h43m3.56628559s) [noise=true , address=145.79.227.40:4058   , piece_id=KRHH2QZKSDPWTSSKYDVDWGKFJD3LPGP6U7KI2PPJ5T3TXYIUGJWA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.695687528 (573h43m3.566770584s) [noise=true , address=77.90.31.19:4857     , piece_id=Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.696135221 (573h43m3.567218276s) [noise=true , address=45.91.116.8:4051     , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.696703733 (573h43m3.567786778s) [noise=true , address=64.49.13.170:4720    , piece_id=HF6X3OT2EPPDE3DWAPWABVZUR3ERDBWXHG3NGNKEWZAD7G5AN4TA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.698418298 (573h43m3.569501343s) [noise=true , address=145.79.227.36:4520   , piece_id=MYJSXUH563JFUW5Q7YUIWL7466G6ASY5WXR3T2PGUFFCTRTOBYJA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.700198829 (573h43m3.571281874s) [noise=true , address=77.90.31.18:4821     , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.701993145 (573h43m3.57307619s) [noise=true , address=193.149.164.143:4699 , piece_id=UDFDCYO25MOJBTMDE6F2WILXE3HLOB75UJZ4LW7ZMNSCOBJI6UJA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.703884687 (573h43m3.574967732s) [noise=true , address=145.79.227.33:4893   , piece_id=XGLX3DFNAY4F4GPGCGZ4ONDJ7NK62SAKX3PBRWHWI74EF3ZFMUSA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.705687109 (573h43m3.576770154s) [noise=true , address=45.91.116.10:4094    , piece_id=6UUIZNTESBBWJMDQ5ZTCHU7S6YKM4GDKO4ZAC6T2FUTPVGFFDS3A, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.707557651 (573h43m3.578640696s) [noise=true , address=145.79.227.43:4164   , piece_id=J6Q4ZDOIIIBA2FXGZZTD6AYS7KGAWSRXXUBAPDPQWYKNU5GNXBBQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:05.168247719 (573h43m5.039330754s) [upload=502975]: Next returned nil segment
Dec 29 13:12:05.168259962 (573h43m5.039343007s) [upload=502975]: Waiting for error group managing segments...
Dec 29 13:12:05.558873127 (573h43m5.429956162s) [noise=true , address=145.79.227.4:4220    , piece_id=SHHJ7EW7DTNOZXVBP6A64VIX7P6AQN62TBCUT3KTH4MIB6UXY3BA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.563024312 (573h43m5.434107347s) [noise=true , address=64.49.13.170:4720    , piece_id=HF6X3OT2EPPDE3DWAPWABVZUR3ERDBWXHG3NGNKEWZAD7G5AN4TA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.564115719 (573h43m5.435198755s) [noise=true , address=145.79.227.36:4520   , piece_id=MYJSXUH563JFUW5Q7YUIWL7466G6ASY5WXR3T2PGUFFCTRTOBYJA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.593873402 (573h43m5.464956447s) [noise=true , address=145.79.227.29:4952   , piece_id=DKYDB2R4DNCGH6T3SLRR2MOZZUPVZZVRL2BKGYOBYXJJVPSZEJ4Q, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.898194604 (573h43m5.769277639s) [noise=true , address=45.91.116.10:4094    , piece_id=6UUIZNTESBBWJMDQ5ZTCHU7S6YKM4GDKO4ZAC6T2FUTPVGFFDS3A, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.912845292 (573h43m5.783928337s) [noise=true , address=77.90.31.19:4857     , piece_id=Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.426532207 (573h43m6.297615252s) [noise=true , address=193.149.164.143:4699 , piece_id=UDFDCYO25MOJBTMDE6F2WILXE3HLOB75UJZ4LW7ZMNSCOBJI6UJA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.449589562 (573h43m6.320672607s) [noise=true , address=77.90.31.15:4711     , piece_id=JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.453186841 (573h43m6.324269876s) [noise=true , address=145.79.227.40:4058   , piece_id=KRHH2QZKSDPWTSSKYDVDWGKFJD3LPGP6U7KI2PPJ5T3TXYIUGJWA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.510769276 (573h43m6.381852321s) [noise=true , address=145.79.227.43:4164   , piece_id=J6Q4ZDOIIIBA2FXGZZTD6AYS7KGAWSRXXUBAPDPQWYKNU5GNXBBQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.518868879 (573h43m6.389951924s) [noise=true , address=145.79.227.33:4893   , piece_id=XGLX3DFNAY4F4GPGCGZ4ONDJ7NK62SAKX3PBRWHWI74EF3ZFMUSA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.519238043 (573h43m6.390321098s) [noise=true , address=45.91.116.6:4981     , piece_id=INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.523211328 (573h43m6.394294373s) [noise=true , address=77.90.31.16:4759     , piece_id=ARC4HRP57U5VQXZTT63QYLXBPZURI5VMJBIQZOFKGLJBQ3K3KGLQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.767869783 (573h43m6.638952818s) [noise=true , address=145.79.227.34:4924   , piece_id=MAQXQTAGXQ7AEPUCESUSUGMHSDMMXS7HHKAEA7B3HW5X66PDUF5A, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.767889431 (573h43m6.638972476s) [seg_pos={0 0}, upload=502975]: Segment reached optimal threshold of 14 pieces.
Dec 29 13:12:06.768102116 (573h43m6.639185161s) [noise=true , address=45.91.116.8:4051     , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1preT33ei4yVXXYVLHhaUbHbiyL14oHFEiof6cCWPig83W6CT3): protocol: expected piece hash; context canceled; EOF
Dec 29 13:12:06.768108257 (573h43m6.639191302s) [noise=true , address=45.91.116.8:4051     , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768219208 (573h43m6.639302254s) [noise=true , address=77.90.31.18:4821     , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH): protocol: expected piece hash; context canceled; EOF
Dec 29 13:12:06.768224789 (573h43m6.639307824s) [noise=true , address=77.90.31.18:4821     , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768384213 (573h43m6.639467258s) [noise=true , address=64.49.13.169:4711    , piece_id=4F4T4YKXMT7NF2C5KBUGJHQJGZL4VCKNKYB266PCVI3UACHJXZ5Q, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection
Dec 29 13:12:06.768399261 (573h43m6.639482296s) [noise=true , address=64.49.13.169:4711    , piece_id=4F4T4YKXMT7NF2C5KBUGJHQJGZL4VCKNKYB266PCVI3UACHJXZ5Q, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768732636 (573h43m6.639815671s) [noise=true , address=64.49.13.169:4709    , piece_id=TBHHX5ES2A6NQRGP25HIC7J7IBWLOJGQ2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:35310->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:35310->64.49.13.169:4709: use of closed network connection
Dec 29 13:12:06.768745460 (573h43m6.639828505s) [noise=true , address=64.49.13.169:4709    , piece_id=TBHHX5ES2A6NQRGP25HIC7J7IBWLOJGQ2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768784586 (573h43m6.639867631s) [seg_pos={0 0}, upload=502975]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 13:12:06.768880217 (573h43m6.639963252s) [upload=502975]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 13:12:06.768884906 (573h43m6.639967941s) [upload=502975]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 13:12:07.067188330 (573h43m6.938271365s) [upload=502975]: Done uploading segments...
Dec 29 13:12:07.067251099 (573h43m6.938334134s) [upload=502975]: Upload finished. err: <nil>
Duration: 54.908827514s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
52 out of 60 is failed (threshold=6) [X.XX.XXX.XX.XXXXXXX.XX.XXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXX.X]
duration: 
5.001284201s
trace: 
https://jaeger.dev.storj.io/trace/56b12eab73796865
FAIL (2025-12-29 13:35:31 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001300251s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
5 out of 60 is failed (threshold=3) [............X....X.X....X.X.................................]
duration: 
18.541945007s
trace: 
https://jaeger.dev.storj.io/trace/312ddb07b2dee82a
FAIL (2025-12-29 13:35:25 UTC) 
orbiter warning: context deadline exceeded
Duration: 18.541960095s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
51 out of 60 is failed (threshold=6) [.X.XX.XXX.XX.XXXXXXX.XX.XXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXX.]
duration: 
3.586250437s
trace: 
https://jaeger.dev.storj.io/trace/50936ed94f0370a4
FAIL (2025-12-29 13:35:07 UTC) 
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.586284782s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
5 out of 60 is failed (threshold=3) [.............X....X.X....X.X................................]
duration: 
26.616358664s
trace: 
https://jaeger.dev.storj.io/trace/312ddb07b2dee82a
FAIL (2025-12-29 13:35:06 UTC) 
orbiter warning: context deadline exceeded
Duration: 26.616388922s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
29 out of 60 is failed (threshold=3) [...X.X.X.XXX.X.X.X.XX..X.XXX.XX..X.XX..XXX...X..X...X...X.XX]
duration: 
30.002710613s
trace: 
https://jaeger.dev.storj.io/trace/577fa942e3b01c43
FAIL (2025-12-29 13:34:59 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 29 13:34:30.554737824 (574h5m30.425820869s) [upload=503298]: Starting upload
Dec 29 13:34:30.554793900 (574h5m30.425876935s) [upload=503298]: Done starting upload
Dec 29 13:34:30.554845740 (574h5m30.425928775s) [upload=503298]: Uploading segments...
Dec 29 13:34:30.555059647 (574h5m30.426142682s) [upload=503298]: Got next segment. Inline: false
Dec 29 13:34:30.555121635 (574h5m30.42620467s) [upload=503298]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 13:34:30.555127165 (574h5m30.4262102s) [upload=503298]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 13:34:30.805697216 (574h5m30.676780261s) [seg_pos={0 0}, upload=503298]: Begin upload segment...
Dec 29 13:34:30.805743525 (574h5m30.67682657s) [seg_pos={0 0}, upload=503298]: Done begin upload segment.
Dec 29 13:34:30.805772790 (574h5m30.676855835s) [noise=true , address=145.79.227.50:4373   , piece_id=KX5NN5YBY6BUJLMB45XT4C6CJCFF366IMXVKNGM6ROXHAPJFLZXQ, seg_pos={0 0}, upload=503298]: Uploading piece...
Dec 29 13:34:30.805854987 (574h5m30.676938042s) [noise=true , address=45.91.116.13:4153    , piece_id=37HNBC62QHNGW2GI2QMYRWS6PI2VIG3HQZZN5TBWKJ4SCTG6TUJQ, seg_pos={0 0}, upload=503298]: Uploading piece...
Dec 29 13:34:30.806046852 (574h5m30.677129887s) [noise=true , address=145.79.227.26:4654   , piece_id=IOKIWIIRWSKSOON2EWB5CUMYYGXN5ZBLPCX7BVA6FIGTHJFWLD5Q, seg_pos={0 0}, upload=503298]: Uploading piece...
Dec 29 13:34:30.806487842 (574h5m30.677570877s) [noise=true , address=45.91.116.11:4118    , piece_id=KLN6SNTZJTPEU2ZJWARDKSA2MWNMMGGLVINW74WIDJ2KB6F6U74A, seg_pos={0 0}, upload=503298]: Uploading piece...
Dec 29 13:34:30.806855823 (574h5m30.677938868s) [noise=true , address=45.91.116.8:4046     , piece_id=EDNAMZ37I5TACP5HDZND5KMYH7LQBW3PEC54QAYRJ67J5CNPGZ7Q, seg_pos={0 0}, upload=503298]: Uploading piece...
Dec 29 13:34:30.807184448 (574h5m30.678267483s) [noise=true , address=145.79.227.11:4732   , piece_id=KJEXJJ6LIO3IMFGRTQT6LE6JX3NKTSS2QBXPCDK4GS3NU2CIAZQA, seg_pos={0 0}, upload=503298]: Uploading piece...
Dec 29 13:34:30.807549263 (574h5m30.678632308s) [noise=true , address=145.79.227.38:4979   , piece_id=NRKTDFZUZVOYZ3LKDRDDYJB4FQL24XKTQ7J6HLHDXMVXPT6S4JUA, seg_pos={0 0}, upload=503298]: Uploading piece...
Dec 29 13:34:30.807855125 (574h5m30.6789382s) [noise=true , address=77.90.31.19:4841     , piece_id=Y4SBNLOG3SPV2IG32RIEC5K6Z4LBEYI3DW7YA2SAS7S2WDAVPVXQ, seg_pos={0 0}, upload=503298]: Uploading piece...
Dec 29 13:34:30.808251420 (574h5m30.679334465s) [noise=true , address=145.79.227.52:4446   , piece_id=52F4WWMJCE6JWRNGTQKVC3GBVTIL72SR6QMEMVEP4WUCFKP7SDBA, seg_pos={0 0}, upload=503298]: Uploading piece...
Dec 29 13:34:30.808595495 (574h5m30.679678531s) [noise=true , address=77.90.31.12:4590     , piece_id=RY37PY55QG6E4CEHGW77EKOCDRVPXKDLWU7KA25HTMML6WZC6LGA, seg_pos={0 0}, upload=503298]: Uploading piece...
Dec 29 13:34:30.808999013 (574h5m30.680082048s) [noise=true , address=145.79.227.22:4511   , piece_id=6HM74AOMDGJ3OERWCJ6QEVTV36HPIR6JHLGIR3YQNKGUBM67BFFQ, seg_pos={0 0}, upload=503298]: Uploading piece...
Dec 29 13:34:30.810432653 (574h5m30.681515688s) [noise=true , address=145.79.227.2:4113    , piece_id=E2PWVHUR7WEBK3TK26WJMKN4BANMVXBHCYX47FSMY7V5GUXB7PYA, seg_pos={0 0}, upload=503298]: Uploading piece...
Dec 29 13:34:30.812130837 (574h5m30.683213872s) [noise=true , address=45.91.116.6:4975     , piece_id=3F35GK6XGKLEKHVL3HIRVNWZGJWKIN46RTKIQXCK7762AXYFERGQ, seg_pos={0 0}, upload=503298]: Uploading piece...
Dec 29 13:34:30.814025275 (574h5m30.68510832s) [noise=true , address=145.79.227.14:4840   , piece_id=DMVPZISJYE4JMLI2D7HZ3GZFFBDM5BIYG7FDYSO4YSI563GMGF7Q, seg_pos={0 0}, upload=503298]: Uploading piece...
Dec 29 13:34:30.815270826 (574h5m30.686353861s) [noise=true , address=145.79.227.46:4601   , piece_id=EKAXJ7F577ZODLYP6WILKUFXC723SIJJKCSKLL6TGYB4WJFAKWSA, seg_pos={0 0}, upload=503298]: Uploading piece...
Dec 29 13:34:30.816406308 (574h5m30.687489343s) [noise=true , address=77.90.31.15:4697     , piece_id=AYZPX7DN7XXHRK4SGPKIUG67OVNOFCZSZFFCMSWLZ3IZVZTC67AA, seg_pos={0 0}, upload=503298]: Uploading piece...
Dec 29 13:34:32.517922063 (574h5m32.389005098s) [upload=503298]: Next returned nil segment
Dec 29 13:34:32.517939686 (574h5m32.389022721s) [upload=503298]: Waiting for error group managing segments...
Dec 29 13:34:32.796270399 (574h5m32.667353444s) [noise=true , address=145.79.227.38:4979   , piece_id=NRKTDFZUZVOYZ3LKDRDDYJB4FQL24XKTQ7J6HLHDXMVXPT6S4JUA, seg_pos={0 0}, upload=503298]: Done uploading piece. err: <nil>
Dec 29 13:34:32.796551454 (574h5m32.667634499s) [noise=true , address=145.79.227.26:4654   , piece_id=IOKIWIIRWSKSOON2EWB5CUMYYGXN5ZBLPCX7BVA6FIGTHJFWLD5Q, seg_pos={0 0}, upload=503298]: Done uploading piece. err: <nil>
Dec 29 13:34:32.797111270 (574h5m32.668194305s) [noise=true , address=145.79.227.22:4511   , piece_id=6HM74AOMDGJ3OERWCJ6QEVTV36HPIR6JHLGIR3YQNKGUBM67BFFQ, seg_pos={0 0}, upload=503298]: Done uploading piece. err: <nil>
Dec 29 13:34:32.801503784 (574h5m32.672586829s) [noise=true , address=145.79.227.2:4113    , piece_id=E2PWVHUR7WEBK3TK26WJMKN4BANMVXBHCYX47FSMY7V5GUXB7PYA, seg_pos={0 0}, upload=503298]: Done uploading piece. err: <nil>
Dec 29 13:34:32.807110980 (574h5m32.678194015s) [noise=true , address=145.79.227.14:4840   , piece_id=DMVPZISJYE4JMLI2D7HZ3GZFFBDM5BIYG7FDYSO4YSI563GMGF7Q, seg_pos={0 0}, upload=503298]: Done uploading piece. err: <nil>
Dec 29 13:34:32.811179297 (574h5m32.682262342s) [noise=true , address=145.79.227.50:4373   , piece_id=KX5NN5YBY6BUJLMB45XT4C6CJCFF366IMXVKNGM6ROXHAPJFLZXQ, seg_pos={0 0}, upload=503298]: Done uploading piece. err: <nil>
Dec 29 13:34:32.818941067 (574h5m32.690024102s) [noise=true , address=145.79.227.11:4732   , piece_id=KJEXJJ6LIO3IMFGRTQT6LE6JX3NKTSS2QBXPCDK4GS3NU2CIAZQA, seg_pos={0 0}, upload=503298]: Done uploading piece. err: <nil>
Dec 29 13:34:32.820337305 (574h5m32.691420351s) [noise=true , address=145.79.227.52:4446   , piece_id=52F4WWMJCE6JWRNGTQKVC3GBVTIL72SR6QMEMVEP4WUCFKP7SDBA, seg_pos={0 0}, upload=503298]: Done uploading piece. err: <nil>
Dec 29 13:34:32.826670365 (574h5m32.69775341s) [noise=true , address=145.79.227.46:4601   , piece_id=EKAXJ7F577ZODLYP6WILKUFXC723SIJJKCSKLL6TGYB4WJFAKWSA, seg_pos={0 0}, upload=503298]: Done uploading piece. err: <nil>
Dec 29 13:34:33.013390778 (574h5m32.884473813s) [noise=true , address=77.90.31.15:4697     , piece_id=AYZPX7DN7XXHRK4SGPKIUG67OVNOFCZSZFFCMSWLZ3IZVZTC67AA, seg_pos={0 0}, upload=503298]: Done uploading piece. err: <nil>
Dec 29 13:34:33.086540485 (574h5m32.95762353s) [noise=true , address=77.90.31.19:4841     , piece_id=Y4SBNLOG3SPV2IG32RIEC5K6Z4LBEYI3DW7YA2SAS7S2WDAVPVXQ, seg_pos={0 0}, upload=503298]: Done uploading piece. err: <nil>
Dec 29 13:34:33.107146002 (574h5m32.978229047s) [noise=true , address=77.90.31.12:4590     , piece_id=RY37PY55QG6E4CEHGW77EKOCDRVPXKDLWU7KA25HTMML6WZC6LGA, seg_pos={0 0}, upload=503298]: Done uploading piece. err: <nil>
Dec 29 13:34:33.116370407 (574h5m32.987453442s) [noise=true , address=45.91.116.11:4118    , piece_id=KLN6SNTZJTPEU2ZJWARDKSA2MWNMMGGLVINW74WIDJ2KB6F6U74A, seg_pos={0 0}, upload=503298]: Done uploading piece. err: <nil>
Dec 29 13:34:33.116389783 (574h5m32.987472818s) [seg_pos={0 0}, upload=503298]: Segment reached optimal threshold of 13 pieces.
Dec 29 13:34:33.116643165 (574h5m32.9877262s) [noise=true , address=45.91.116.6:4975     , piece_id=3F35GK6XGKLEKHVL3HIRVNWZGJWKIN46RTKIQXCK7762AXYFERGQ, seg_pos={0 0}, upload=503298]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF
Dec 29 13:34:33.116650009 (574h5m32.987733044s) [noise=true , address=45.91.116.6:4975     , piece_id=3F35GK6XGKLEKHVL3HIRVNWZGJWKIN46RTKIQXCK7762AXYFERGQ, seg_pos={0 0}, upload=503298]: Long tail cancellation due to optimal threshold
Dec 29 13:34:33.116802689 (574h5m32.987885724s) [noise=true , address=45.91.116.13:4153    , piece_id=37HNBC62QHNGW2GI2QMYRWS6PI2VIG3HQZZN5TBWKJ4SCTG6TUJQ, seg_pos={0 0}, upload=503298]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF
Dec 29 13:34:33.116808139 (574h5m32.987891175s) [noise=true , address=45.91.116.13:4153    , piece_id=37HNBC62QHNGW2GI2QMYRWS6PI2VIG3HQZZN5TBWKJ4SCTG6TUJQ, seg_pos={0 0}, upload=503298]: Long tail cancellation due to optimal threshold
Dec 29 13:34:33.117012178 (574h5m32.988095213s) [noise=true , address=45.91.116.8:4046     , piece_id=EDNAMZ37I5TACP5HDZND5KMYH7LQBW3PEC54QAYRJ67J5CNPGZ7Q, seg_pos={0 0}, upload=503298]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF
Dec 29 13:34:33.117032427 (574h5m32.988115492s) [noise=true , address=45.91.116.8:4046     , piece_id=EDNAMZ37I5TACP5HDZND5KMYH7LQBW3PEC54QAYRJ67J5CNPGZ7Q, seg_pos={0 0}, upload=503298]: Long tail cancellation due to optimal threshold
Dec 29 13:34:33.117059378 (574h5m32.988142413s) [seg_pos={0 0}, upload=503298]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 13:34:33.117153457 (574h5m32.988236492s) [upload=503298]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 13:34:33.117158867 (574h5m32.988241902s) [upload=503298]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 13:34:33.412991256 (574h5m33.284074291s) [upload=503298]: Done uploading segments...
Dec 29 13:34:33.413022756 (574h5m33.284105791s) [upload=503298]: Upload finished. err: <nil>
Duration: 30.00273042s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
4 out of 60 is failed (threshold=3) [.....X.....X.....................XX.........................]
duration: 
55.049870099s
trace: 
https://jaeger.dev.storj.io/trace/77d26141ea11d228
FAIL (2025-12-29 13:34:37 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12J2weU7Bmsr3RdQQ9QMuadT96XReBkyxTeaLJ2SEqVGGFX1PWu, Piece ID: JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ) context deadline exceeded; piecestore: (Node ID: 125sm6iHV94mJFLWrKeyRMH4b1qSMo32nDuW8uQr71xUL7Cema2, Piece ID: INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA) context deadline exceeded; piecestore: (Node ID: 12i57FF97oEddYLZpNxSq5or9WLEquL8ZW1cVUuW4AThBbDqcCL, Piece ID: Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA) context deadline exceeded
======LOGS======
Q2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.692912334 (573h43m3.563995379s) [noise=true , address=77.90.31.16:4759     , piece_id=ARC4HRP57U5VQXZTT63QYLXBPZURI5VMJBIQZOFKGLJBQ3K3KGLQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.693322595 (573h43m3.56440564s) [noise=true , address=77.90.31.15:4711     , piece_id=JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.693739649 (573h43m3.564822694s) [noise=true , address=45.91.116.6:4981     , piece_id=INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.694315887 (573h43m3.565398942s) [noise=true , address=145.79.227.4:4220    , piece_id=SHHJ7EW7DTNOZXVBP6A64VIX7P6AQN62TBCUT3KTH4MIB6UXY3BA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.694765222 (573h43m3.565848267s) [noise=true , address=145.79.227.29:4952   , piece_id=DKYDB2R4DNCGH6T3SLRR2MOZZUPVZZVRL2BKGYOBYXJJVPSZEJ4Q, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.695202545 (573h43m3.56628559s) [noise=true , address=145.79.227.40:4058   , piece_id=KRHH2QZKSDPWTSSKYDVDWGKFJD3LPGP6U7KI2PPJ5T3TXYIUGJWA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.695687528 (573h43m3.566770584s) [noise=true , address=77.90.31.19:4857     , piece_id=Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.696135221 (573h43m3.567218276s) [noise=true , address=45.91.116.8:4051     , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.696703733 (573h43m3.567786778s) [noise=true , address=64.49.13.170:4720    , piece_id=HF6X3OT2EPPDE3DWAPWABVZUR3ERDBWXHG3NGNKEWZAD7G5AN4TA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.698418298 (573h43m3.569501343s) [noise=true , address=145.79.227.36:4520   , piece_id=MYJSXUH563JFUW5Q7YUIWL7466G6ASY5WXR3T2PGUFFCTRTOBYJA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.700198829 (573h43m3.571281874s) [noise=true , address=77.90.31.18:4821     , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.701993145 (573h43m3.57307619s) [noise=true , address=193.149.164.143:4699 , piece_id=UDFDCYO25MOJBTMDE6F2WILXE3HLOB75UJZ4LW7ZMNSCOBJI6UJA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.703884687 (573h43m3.574967732s) [noise=true , address=145.79.227.33:4893   , piece_id=XGLX3DFNAY4F4GPGCGZ4ONDJ7NK62SAKX3PBRWHWI74EF3ZFMUSA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.705687109 (573h43m3.576770154s) [noise=true , address=45.91.116.10:4094    , piece_id=6UUIZNTESBBWJMDQ5ZTCHU7S6YKM4GDKO4ZAC6T2FUTPVGFFDS3A, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.707557651 (573h43m3.578640696s) [noise=true , address=145.79.227.43:4164   , piece_id=J6Q4ZDOIIIBA2FXGZZTD6AYS7KGAWSRXXUBAPDPQWYKNU5GNXBBQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:05.168247719 (573h43m5.039330754s) [upload=502975]: Next returned nil segment
Dec 29 13:12:05.168259962 (573h43m5.039343007s) [upload=502975]: Waiting for error group managing segments...
Dec 29 13:12:05.558873127 (573h43m5.429956162s) [noise=true , address=145.79.227.4:4220    , piece_id=SHHJ7EW7DTNOZXVBP6A64VIX7P6AQN62TBCUT3KTH4MIB6UXY3BA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.563024312 (573h43m5.434107347s) [noise=true , address=64.49.13.170:4720    , piece_id=HF6X3OT2EPPDE3DWAPWABVZUR3ERDBWXHG3NGNKEWZAD7G5AN4TA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.564115719 (573h43m5.435198755s) [noise=true , address=145.79.227.36:4520   , piece_id=MYJSXUH563JFUW5Q7YUIWL7466G6ASY5WXR3T2PGUFFCTRTOBYJA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.593873402 (573h43m5.464956447s) [noise=true , address=145.79.227.29:4952   , piece_id=DKYDB2R4DNCGH6T3SLRR2MOZZUPVZZVRL2BKGYOBYXJJVPSZEJ4Q, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.898194604 (573h43m5.769277639s) [noise=true , address=45.91.116.10:4094    , piece_id=6UUIZNTESBBWJMDQ5ZTCHU7S6YKM4GDKO4ZAC6T2FUTPVGFFDS3A, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.912845292 (573h43m5.783928337s) [noise=true , address=77.90.31.19:4857     , piece_id=Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.426532207 (573h43m6.297615252s) [noise=true , address=193.149.164.143:4699 , piece_id=UDFDCYO25MOJBTMDE6F2WILXE3HLOB75UJZ4LW7ZMNSCOBJI6UJA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.449589562 (573h43m6.320672607s) [noise=true , address=77.90.31.15:4711     , piece_id=JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.453186841 (573h43m6.324269876s) [noise=true , address=145.79.227.40:4058   , piece_id=KRHH2QZKSDPWTSSKYDVDWGKFJD3LPGP6U7KI2PPJ5T3TXYIUGJWA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.510769276 (573h43m6.381852321s) [noise=true , address=145.79.227.43:4164   , piece_id=J6Q4ZDOIIIBA2FXGZZTD6AYS7KGAWSRXXUBAPDPQWYKNU5GNXBBQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.518868879 (573h43m6.389951924s) [noise=true , address=145.79.227.33:4893   , piece_id=XGLX3DFNAY4F4GPGCGZ4ONDJ7NK62SAKX3PBRWHWI74EF3ZFMUSA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.519238043 (573h43m6.390321098s) [noise=true , address=45.91.116.6:4981     , piece_id=INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.523211328 (573h43m6.394294373s) [noise=true , address=77.90.31.16:4759     , piece_id=ARC4HRP57U5VQXZTT63QYLXBPZURI5VMJBIQZOFKGLJBQ3K3KGLQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.767869783 (573h43m6.638952818s) [noise=true , address=145.79.227.34:4924   , piece_id=MAQXQTAGXQ7AEPUCESUSUGMHSDMMXS7HHKAEA7B3HW5X66PDUF5A, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.767889431 (573h43m6.638972476s) [seg_pos={0 0}, upload=502975]: Segment reached optimal threshold of 14 pieces.
Dec 29 13:12:06.768102116 (573h43m6.639185161s) [noise=true , address=45.91.116.8:4051     , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1preT33ei4yVXXYVLHhaUbHbiyL14oHFEiof6cCWPig83W6CT3): protocol: expected piece hash; context canceled; EOF
Dec 29 13:12:06.768108257 (573h43m6.639191302s) [noise=true , address=45.91.116.8:4051     , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768219208 (573h43m6.639302254s) [noise=true , address=77.90.31.18:4821     , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH): protocol: expected piece hash; context canceled; EOF
Dec 29 13:12:06.768224789 (573h43m6.639307824s) [noise=true , address=77.90.31.18:4821     , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768384213 (573h43m6.639467258s) [noise=true , address=64.49.13.169:4711    , piece_id=4F4T4YKXMT7NF2C5KBUGJHQJGZL4VCKNKYB266PCVI3UACHJXZ5Q, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection
Dec 29 13:12:06.768399261 (573h43m6.639482296s) [noise=true , address=64.49.13.169:4711    , piece_id=4F4T4YKXMT7NF2C5KBUGJHQJGZL4VCKNKYB266PCVI3UACHJXZ5Q, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768732636 (573h43m6.639815671s) [noise=true , address=64.49.13.169:4709    , piece_id=TBHHX5ES2A6NQRGP25HIC7J7IBWLOJGQ2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:35310->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:35310->64.49.13.169:4709: use of closed network connection
Dec 29 13:12:06.768745460 (573h43m6.639828505s) [noise=true , address=64.49.13.169:4709    , piece_id=TBHHX5ES2A6NQRGP25HIC7J7IBWLOJGQ2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768784586 (573h43m6.639867631s) [seg_pos={0 0}, upload=502975]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 13:12:06.768880217 (573h43m6.639963252s) [upload=502975]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 13:12:06.768884906 (573h43m6.639967941s) [upload=502975]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 13:12:07.067188330 (573h43m6.938271365s) [upload=502975]: Done uploading segments...
Dec 29 13:12:07.067251099 (573h43m6.938334134s) [upload=502975]: Upload finished. err: <nil>
Duration: 55.049882622s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
5 out of 60 is failed (threshold=3) [..............X....X.X....X.X...............................]
duration: 
18.421100692s
trace: 
https://jaeger.dev.storj.io/trace/312ddb07b2dee82a
FAIL (2025-12-29 13:34:33 UTC) 
orbiter warning: context deadline exceeded
Duration: 18.421130609s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
29 out of 60 is failed (threshold=3) [X...X.X.X.XXX.X.X.X.XX..X.XXX.XX..X.XX..XXX...X..X...X...X.X]
duration: 
30.003632837s
trace: 
https://jaeger.dev.storj.io/trace/575e34d5da8430f0
FAIL (2025-12-29 13:34:25 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: QBCBTRLCCYN66LFEV2FJ6BCZKQCLYCAPWBWF62UKJZTK3UPXF2XQ) context deadline exceeded; piecestore: (Node ID: 1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, Piece ID: GXBZF4BBHPZT5TXDSTTSS6IX6457OIDJUM7FPF5NIPTYUMYDTPXA) context deadline exceeded
======LOGS======
Dec 29 13:33:56.502117331 (574h4m56.373200366s) [upload=503289]: Starting upload
Dec 29 13:33:56.502300750 (574h4m56.373383785s) [upload=503289]: Done starting upload
Dec 29 13:33:56.502354973 (574h4m56.373438018s) [upload=503289]: Uploading segments...
Dec 29 13:33:56.502563300 (574h4m56.373646345s) [upload=503289]: Got next segment. Inline: false
Dec 29 13:33:56.502602926 (574h4m56.373685961s) [upload=503289]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 13:33:56.502607394 (574h4m56.373690429s) [upload=503289]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 13:33:56.752260640 (574h4m56.623343675s) [seg_pos={0 0}, upload=503289]: Begin upload segment...
Dec 29 13:33:56.752314542 (574h4m56.623397578s) [seg_pos={0 0}, upload=503289]: Done begin upload segment.
Dec 29 13:33:56.752350130 (574h4m56.623433175s) [noise=true , address=45.91.116.9:4263     , piece_id=IUHPVAWQHGD7LWYUQ2TY3BTX7UVSSCP74TD7URMO5MC77XOUJPDQ, seg_pos={0 0}, upload=503289]: Uploading piece...
Dec 29 13:33:56.752405827 (574h4m56.623488872s) [noise=true , address=145.79.227.3:4150    , piece_id=B2VSL7INHQED4U323YHD56HYA7HFR23YFSENUPZDJCD3Z4TKM3RA, seg_pos={0 0}, upload=503289]: Uploading piece...
Dec 29 13:33:56.753344314 (574h4m56.624427349s) [noise=true , address=145.79.227.16:4909   , piece_id=ZJBJ6SR4CDVS2DJ2X2JTYI35TUEC67CD5VH3OPYOUDEOV737FVYA, seg_pos={0 0}, upload=503289]: Uploading piece...
Dec 29 13:33:56.753422342 (574h4m56.624505387s) [noise=true , address=45.91.116.7:4009     , piece_id=HL2Y7YAPI4PADGTQHPII3XL2QRSN7YH4YCEW6JEH5PFHY7ENGTFA, seg_pos={0 0}, upload=503289]: Uploading piece...
Dec 29 13:33:56.753850467 (574h4m56.624933502s) [noise=true , address=145.79.227.28:4725   , piece_id=V7VVBQCT33LB6JHVMGMPQCLJCOCNFTRATVPQSFOOTLIDVZCKS3XQ, seg_pos={0 0}, upload=503289]: Uploading piece...
Dec 29 13:33:56.754185495 (574h4m56.62526854s) [noise=true , address=145.79.227.15:4874   , piece_id=CW2MPRZIH5CLV5LIMT77GDC5VEBCPCTDMX2B7DTPZISVBMDOPGBA, seg_pos={0 0}, upload=503289]: Uploading piece...
Dec 29 13:33:56.754355279 (574h4m56.625438314s) [noise=true , address=145.79.227.45:4231   , piece_id=W6FXHOSXLJNTEKRMR2RA4FM7QLI2SDI4ZMD7VGLCMXOEDFF2JAQA, seg_pos={0 0}, upload=503289]: Uploading piece...
Dec 29 13:33:56.754523528 (574h4m56.625606563s) [noise=true , address=45.91.116.8:4046     , piece_id=QBCBTRLCCYN66LFEV2FJ6BCZKQCLYCAPWBWF62UKJZTK3UPXF2XQ, seg_pos={0 0}, upload=503289]: Uploading piece...
Dec 29 13:33:56.754703051 (574h4m56.625786086s) [noise=true , address=145.79.227.11:4732   , piece_id=OPWCTCXVHHFWWJRAQMYHSGYUC2IXBS44LYKFROM6JF2IVLQ2WNLQ, seg_pos={0 0}, upload=503289]: Uploading piece...
Dec 29 13:33:56.754879046 (574h4m56.625962081s) [noise=true , address=77.90.31.11:4553     , piece_id=GXBZF4BBHPZT5TXDSTTSS6IX6457OIDJUM7FPF5NIPTYUMYDTPXA, seg_pos={0 0}, upload=503289]: Uploading piece...
Dec 29 13:33:56.755080549 (574h4m56.626163584s) [noise=true , address=145.79.227.32:4835   , piece_id=DA76HWORZZOQRW5I2OTM7VOU32IH2RGPYSKFD6AEGB72DVNWMKPQ, seg_pos={0 0}, upload=503289]: Uploading piece...
Dec 29 13:33:56.755233250 (574h4m56.626316285s) [noise=true , address=145.79.227.48:4301   , piece_id=U6W4XOZ5JHDHU3N3LJGH4EPDMXELKYPF6YVQ3QIMXP5TIWYVOO5Q, seg_pos={0 0}, upload=503289]: Uploading piece...
Dec 29 13:33:56.755428181 (574h4m56.626511216s) [noise=true , address=145.79.227.18:4007   , piece_id=MSBUQNU4KXZPSRSS5VJQFXZ2DNCISL7TVT3STVFWLFPVYJJ6ZUUA, seg_pos={0 0}, upload=503289]: Uploading piece...
Dec 29 13:33:56.755814767 (574h4m56.626897802s) [noise=true , address=145.79.227.37:4565   , piece_id=Z5YRSTZVNFM4XDCGRVO2WPJ4OGT5LEXUS7X7VTVXUVKNVE36JTZA, seg_pos={0 0}, upload=503289]: Uploading piece...
Dec 29 13:33:56.756200482 (574h4m56.627283517s) [noise=true , address=145.79.227.40:4051   , piece_id=JWFSJMR6SRAA5VVVAMMGWA6SBVFUINAHWYGIY4JJPSXEAQKOYX3Q, seg_pos={0 0}, upload=503289]: Uploading piece...
Dec 29 13:33:56.756588972 (574h4m56.627672007s) [noise=true , address=45.91.116.4:4939     , piece_id=MSFVB7J2WLUOLPHSIAUZZYQLXKBA2DLCNCQMUOMZBRNQU5BLT2TA, seg_pos={0 0}, upload=503289]: Uploading piece...
Dec 29 13:33:58.455909000 (574h4m58.326992035s) [upload=503289]: Next returned nil segment
Dec 29 13:33:58.455926353 (574h4m58.327009418s) [upload=503289]: Waiting for error group managing segments...
Dec 29 13:33:58.731853289 (574h4m58.602936334s) [noise=true , address=145.79.227.11:4732   , piece_id=OPWCTCXVHHFWWJRAQMYHSGYUC2IXBS44LYKFROM6JF2IVLQ2WNLQ, seg_pos={0 0}, upload=503289]: Done uploading piece. err: <nil>
Dec 29 13:33:58.733845382 (574h4m58.604928427s) [noise=true , address=145.79.227.15:4874   , piece_id=CW2MPRZIH5CLV5LIMT77GDC5VEBCPCTDMX2B7DTPZISVBMDOPGBA, seg_pos={0 0}, upload=503289]: Done uploading piece. err: <nil>
Dec 29 13:33:58.737840771 (574h4m58.608923816s) [noise=true , address=145.79.227.3:4150    , piece_id=B2VSL7INHQED4U323YHD56HYA7HFR23YFSENUPZDJCD3Z4TKM3RA, seg_pos={0 0}, upload=503289]: Done uploading piece. err: <nil>
Dec 29 13:33:58.741407522 (574h4m58.612490557s) [noise=true , address=145.79.227.16:4909   , piece_id=ZJBJ6SR4CDVS2DJ2X2JTYI35TUEC67CD5VH3OPYOUDEOV737FVYA, seg_pos={0 0}, upload=503289]: Done uploading piece. err: <nil>
Dec 29 13:33:58.745871762 (574h4m58.616954807s) [noise=true , address=145.79.227.32:4835   , piece_id=DA76HWORZZOQRW5I2OTM7VOU32IH2RGPYSKFD6AEGB72DVNWMKPQ, seg_pos={0 0}, upload=503289]: Done uploading piece. err: <nil>
Dec 29 13:33:58.758027348 (574h4m58.629110383s) [noise=true , address=145.79.227.45:4231   , piece_id=W6FXHOSXLJNTEKRMR2RA4FM7QLI2SDI4ZMD7VGLCMXOEDFF2JAQA, seg_pos={0 0}, upload=503289]: Done uploading piece. err: <nil>
Dec 29 13:33:58.763312522 (574h4m58.634395557s) [noise=true , address=145.79.227.28:4725   , piece_id=V7VVBQCT33LB6JHVMGMPQCLJCOCNFTRATVPQSFOOTLIDVZCKS3XQ, seg_pos={0 0}, upload=503289]: Done uploading piece. err: <nil>
Dec 29 13:33:58.765006638 (574h4m58.636089683s) [noise=true , address=145.79.227.37:4565   , piece_id=Z5YRSTZVNFM4XDCGRVO2WPJ4OGT5LEXUS7X7VTVXUVKNVE36JTZA, seg_pos={0 0}, upload=503289]: Done uploading piece. err: <nil>
Dec 29 13:33:58.767789186 (574h4m58.638872231s) [noise=true , address=145.79.227.48:4301   , piece_id=U6W4XOZ5JHDHU3N3LJGH4EPDMXELKYPF6YVQ3QIMXP5TIWYVOO5Q, seg_pos={0 0}, upload=503289]: Done uploading piece. err: <nil>
Dec 29 13:33:58.768026318 (574h4m58.639109353s) [noise=true , address=145.79.227.18:4007   , piece_id=MSBUQNU4KXZPSRSS5VJQFXZ2DNCISL7TVT3STVFWLFPVYJJ6ZUUA, seg_pos={0 0}, upload=503289]: Done uploading piece. err: <nil>
Dec 29 13:33:59.061738609 (574h4m58.932821654s) [noise=true , address=77.90.31.11:4553     , piece_id=GXBZF4BBHPZT5TXDSTTSS6IX6457OIDJUM7FPF5NIPTYUMYDTPXA, seg_pos={0 0}, upload=503289]: Done uploading piece. err: <nil>
Dec 29 13:33:59.106214779 (574h4m58.977297814s) [noise=true , address=45.91.116.8:4046     , piece_id=QBCBTRLCCYN66LFEV2FJ6BCZKQCLYCAPWBWF62UKJZTK3UPXF2XQ, seg_pos={0 0}, upload=503289]: Done uploading piece. err: <nil>
Dec 29 13:33:59.130785245 (574h4m59.00186829s) [noise=true , address=45.91.116.9:4263     , piece_id=IUHPVAWQHGD7LWYUQ2TY3BTX7UVSSCP74TD7URMO5MC77XOUJPDQ, seg_pos={0 0}, upload=503289]: Done uploading piece. err: <nil>
Dec 29 13:33:59.130807758 (574h4m59.001890803s) [seg_pos={0 0}, upload=503289]: Segment reached optimal threshold of 13 pieces.
Dec 29 13:33:59.131036102 (574h4m59.002119137s) [noise=true , address=45.91.116.7:4009     , piece_id=HL2Y7YAPI4PADGTQHPII3XL2QRSN7YH4YCEW6JEH5PFHY7ENGTFA, seg_pos={0 0}, upload=503289]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Dec 29 13:33:59.131044458 (574h4m59.002127493s) [noise=true , address=45.91.116.7:4009     , piece_id=HL2Y7YAPI4PADGTQHPII3XL2QRSN7YH4YCEW6JEH5PFHY7ENGTFA, seg_pos={0 0}, upload=503289]: Long tail cancellation due to optimal threshold
Dec 29 13:33:59.131194955 (574h4m59.002278s) [noise=true , address=145.79.227.40:4051   , piece_id=JWFSJMR6SRAA5VVVAMMGWA6SBVFUINAHWYGIY4JJPSXEAQKOYX3Q, seg_pos={0 0}, upload=503289]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF
Dec 29 13:33:59.131206797 (574h4m59.002289832s) [noise=true , address=145.79.227.40:4051   , piece_id=JWFSJMR6SRAA5VVVAMMGWA6SBVFUINAHWYGIY4JJPSXEAQKOYX3Q, seg_pos={0 0}, upload=503289]: Long tail cancellation due to optimal threshold
Dec 29 13:33:59.131237305 (574h4m59.002320341s) [noise=true , address=45.91.116.4:4939     , piece_id=MSFVB7J2WLUOLPHSIAUZZYQLXKBA2DLCNCQMUOMZBRNQU5BLT2TA, seg_pos={0 0}, upload=503289]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:45382->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:45382->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45382->45.91.116.4:4939: use of closed network connection
Dec 29 13:33:59.131787143 (574h4m59.002870318s) [noise=true , address=45.91.116.4:4939     , piece_id=MSFVB7J2WLUOLPHSIAUZZYQLXKBA2DLCNCQMUOMZBRNQU5BLT2TA, seg_pos={0 0}, upload=503289]: Long tail cancellation due to optimal threshold
Dec 29 13:33:59.131860693 (574h4m59.002943738s) [seg_pos={0 0}, upload=503289]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 13:33:59.132105930 (574h4m59.003188975s) [upload=503289]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 13:33:59.132114976 (574h4m59.003198012s) [upload=503289]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 13:33:59.424726282 (574h4m59.295809317s) [upload=503289]: Done uploading segments...
Dec 29 13:33:59.424745338 (574h4m59.295828373s) [upload=503289]: Upload finished. err: <nil>
Duration: 30.003653758s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
52 out of 60 is failed (threshold=6) [X.X.XX.XXX.XX.XXXXXXX.XX.XXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXX]
duration: 
5.001584734s
trace: 
https://jaeger.dev.storj.io/trace/50936ed94f0370a4
FAIL (2025-12-29 13:34:19 UTC) 
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001643877s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
5 out of 60 is failed (threshold=3) [...............X....X.X....X.X..............................]
duration: 
19.846299992s
trace: 
https://jaeger.dev.storj.io/trace/312ddb07b2dee82a
FAIL (2025-12-29 13:34:13 UTC) 
orbiter warning: context deadline exceeded
Duration: 19.846370336s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
4 out of 60 is failed (threshold=3) [......X.....X.....................XX........................]
duration: 
45.810271041s
trace: 
https://jaeger.dev.storj.io/trace/77d26141ea11d228
FAIL (2025-12-29 13:33:42 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12J2weU7Bmsr3RdQQ9QMuadT96XReBkyxTeaLJ2SEqVGGFX1PWu, Piece ID: JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ) context deadline exceeded; piecestore: (Node ID: 125sm6iHV94mJFLWrKeyRMH4b1qSMo32nDuW8uQr71xUL7Cema2, Piece ID: INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA) context deadline exceeded; piecestore: (Node ID: 12i57FF97oEddYLZpNxSq5or9WLEquL8ZW1cVUuW4AThBbDqcCL, Piece ID: Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA) context deadline exceeded
======LOGS======
Q2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.692912334 (573h43m3.563995379s) [noise=true , address=77.90.31.16:4759     , piece_id=ARC4HRP57U5VQXZTT63QYLXBPZURI5VMJBIQZOFKGLJBQ3K3KGLQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.693322595 (573h43m3.56440564s) [noise=true , address=77.90.31.15:4711     , piece_id=JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.693739649 (573h43m3.564822694s) [noise=true , address=45.91.116.6:4981     , piece_id=INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.694315887 (573h43m3.565398942s) [noise=true , address=145.79.227.4:4220    , piece_id=SHHJ7EW7DTNOZXVBP6A64VIX7P6AQN62TBCUT3KTH4MIB6UXY3BA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.694765222 (573h43m3.565848267s) [noise=true , address=145.79.227.29:4952   , piece_id=DKYDB2R4DNCGH6T3SLRR2MOZZUPVZZVRL2BKGYOBYXJJVPSZEJ4Q, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.695202545 (573h43m3.56628559s) [noise=true , address=145.79.227.40:4058   , piece_id=KRHH2QZKSDPWTSSKYDVDWGKFJD3LPGP6U7KI2PPJ5T3TXYIUGJWA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.695687528 (573h43m3.566770584s) [noise=true , address=77.90.31.19:4857     , piece_id=Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.696135221 (573h43m3.567218276s) [noise=true , address=45.91.116.8:4051     , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.696703733 (573h43m3.567786778s) [noise=true , address=64.49.13.170:4720    , piece_id=HF6X3OT2EPPDE3DWAPWABVZUR3ERDBWXHG3NGNKEWZAD7G5AN4TA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.698418298 (573h43m3.569501343s) [noise=true , address=145.79.227.36:4520   , piece_id=MYJSXUH563JFUW5Q7YUIWL7466G6ASY5WXR3T2PGUFFCTRTOBYJA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.700198829 (573h43m3.571281874s) [noise=true , address=77.90.31.18:4821     , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.701993145 (573h43m3.57307619s) [noise=true , address=193.149.164.143:4699 , piece_id=UDFDCYO25MOJBTMDE6F2WILXE3HLOB75UJZ4LW7ZMNSCOBJI6UJA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.703884687 (573h43m3.574967732s) [noise=true , address=145.79.227.33:4893   , piece_id=XGLX3DFNAY4F4GPGCGZ4ONDJ7NK62SAKX3PBRWHWI74EF3ZFMUSA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.705687109 (573h43m3.576770154s) [noise=true , address=45.91.116.10:4094    , piece_id=6UUIZNTESBBWJMDQ5ZTCHU7S6YKM4GDKO4ZAC6T2FUTPVGFFDS3A, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.707557651 (573h43m3.578640696s) [noise=true , address=145.79.227.43:4164   , piece_id=J6Q4ZDOIIIBA2FXGZZTD6AYS7KGAWSRXXUBAPDPQWYKNU5GNXBBQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:05.168247719 (573h43m5.039330754s) [upload=502975]: Next returned nil segment
Dec 29 13:12:05.168259962 (573h43m5.039343007s) [upload=502975]: Waiting for error group managing segments...
Dec 29 13:12:05.558873127 (573h43m5.429956162s) [noise=true , address=145.79.227.4:4220    , piece_id=SHHJ7EW7DTNOZXVBP6A64VIX7P6AQN62TBCUT3KTH4MIB6UXY3BA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.563024312 (573h43m5.434107347s) [noise=true , address=64.49.13.170:4720    , piece_id=HF6X3OT2EPPDE3DWAPWABVZUR3ERDBWXHG3NGNKEWZAD7G5AN4TA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.564115719 (573h43m5.435198755s) [noise=true , address=145.79.227.36:4520   , piece_id=MYJSXUH563JFUW5Q7YUIWL7466G6ASY5WXR3T2PGUFFCTRTOBYJA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.593873402 (573h43m5.464956447s) [noise=true , address=145.79.227.29:4952   , piece_id=DKYDB2R4DNCGH6T3SLRR2MOZZUPVZZVRL2BKGYOBYXJJVPSZEJ4Q, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.898194604 (573h43m5.769277639s) [noise=true , address=45.91.116.10:4094    , piece_id=6UUIZNTESBBWJMDQ5ZTCHU7S6YKM4GDKO4ZAC6T2FUTPVGFFDS3A, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.912845292 (573h43m5.783928337s) [noise=true , address=77.90.31.19:4857     , piece_id=Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.426532207 (573h43m6.297615252s) [noise=true , address=193.149.164.143:4699 , piece_id=UDFDCYO25MOJBTMDE6F2WILXE3HLOB75UJZ4LW7ZMNSCOBJI6UJA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.449589562 (573h43m6.320672607s) [noise=true , address=77.90.31.15:4711     , piece_id=JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.453186841 (573h43m6.324269876s) [noise=true , address=145.79.227.40:4058   , piece_id=KRHH2QZKSDPWTSSKYDVDWGKFJD3LPGP6U7KI2PPJ5T3TXYIUGJWA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.510769276 (573h43m6.381852321s) [noise=true , address=145.79.227.43:4164   , piece_id=J6Q4ZDOIIIBA2FXGZZTD6AYS7KGAWSRXXUBAPDPQWYKNU5GNXBBQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.518868879 (573h43m6.389951924s) [noise=true , address=145.79.227.33:4893   , piece_id=XGLX3DFNAY4F4GPGCGZ4ONDJ7NK62SAKX3PBRWHWI74EF3ZFMUSA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.519238043 (573h43m6.390321098s) [noise=true , address=45.91.116.6:4981     , piece_id=INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.523211328 (573h43m6.394294373s) [noise=true , address=77.90.31.16:4759     , piece_id=ARC4HRP57U5VQXZTT63QYLXBPZURI5VMJBIQZOFKGLJBQ3K3KGLQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.767869783 (573h43m6.638952818s) [noise=true , address=145.79.227.34:4924   , piece_id=MAQXQTAGXQ7AEPUCESUSUGMHSDMMXS7HHKAEA7B3HW5X66PDUF5A, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.767889431 (573h43m6.638972476s) [seg_pos={0 0}, upload=502975]: Segment reached optimal threshold of 14 pieces.
Dec 29 13:12:06.768102116 (573h43m6.639185161s) [noise=true , address=45.91.116.8:4051     , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1preT33ei4yVXXYVLHhaUbHbiyL14oHFEiof6cCWPig83W6CT3): protocol: expected piece hash; context canceled; EOF
Dec 29 13:12:06.768108257 (573h43m6.639191302s) [noise=true , address=45.91.116.8:4051     , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768219208 (573h43m6.639302254s) [noise=true , address=77.90.31.18:4821     , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH): protocol: expected piece hash; context canceled; EOF
Dec 29 13:12:06.768224789 (573h43m6.639307824s) [noise=true , address=77.90.31.18:4821     , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768384213 (573h43m6.639467258s) [noise=true , address=64.49.13.169:4711    , piece_id=4F4T4YKXMT7NF2C5KBUGJHQJGZL4VCKNKYB266PCVI3UACHJXZ5Q, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection
Dec 29 13:12:06.768399261 (573h43m6.639482296s) [noise=true , address=64.49.13.169:4711    , piece_id=4F4T4YKXMT7NF2C5KBUGJHQJGZL4VCKNKYB266PCVI3UACHJXZ5Q, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768732636 (573h43m6.639815671s) [noise=true , address=64.49.13.169:4709    , piece_id=TBHHX5ES2A6NQRGP25HIC7J7IBWLOJGQ2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:35310->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:35310->64.49.13.169:4709: use of closed network connection
Dec 29 13:12:06.768745460 (573h43m6.639828505s) [noise=true , address=64.49.13.169:4709    , piece_id=TBHHX5ES2A6NQRGP25HIC7J7IBWLOJGQ2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768784586 (573h43m6.639867631s) [seg_pos={0 0}, upload=502975]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 13:12:06.768880217 (573h43m6.639963252s) [upload=502975]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 13:12:06.768884906 (573h43m6.639967941s) [upload=502975]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 13:12:07.067188330 (573h43m6.938271365s) [upload=502975]: Done uploading segments...
Dec 29 13:12:07.067251099 (573h43m6.938334134s) [upload=502975]: Upload finished. err: <nil>
Duration: 45.810295777s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
52 out of 60 is failed (threshold=6) [XX.X.XX.XXX.XX.XXXXXXX.XX.XXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXX]
duration: 
5.027632497s
trace: 
https://jaeger.dev.storj.io/trace/2e359418080a25b2
FAIL (2025-12-29 13:33:38 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.027798914s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
29 out of 60 is failed (threshold=3) [XX...X.X.X.XXX.X.X.X.XX..X.XXX.XX..X.XX..XXX...X..X...X...X.]
duration: 
6.605299132s
trace: 
https://jaeger.dev.storj.io/trace/4b55f632d9c143f
FAIL (2025-12-29 13:33:31 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: GKDOJZSB7BSLWRCXRUJAEUFHTKW3OSXIL2TRM4WTV6F5XD66Q2MQ) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: MLZ6HV66PLZEHMANEUDUS7QKK5CQDYYVXUYGKKUIHOEVZD75V5FA) context deadline exceeded
======LOGS======
Dec 29 13:32:54.921929839 (574h3m54.793012874s) [upload=503272]: Starting upload
Dec 29 13:32:54.921991305 (574h3m54.79307434s) [upload=503272]: Done starting upload
Dec 29 13:32:54.922216405 (574h3m54.79329944s) [upload=503272]: Uploading segments...
Dec 29 13:32:54.922257413 (574h3m54.793340458s) [upload=503272]: Got next segment. Inline: false
Dec 29 13:32:54.922295084 (574h3m54.793378119s) [upload=503272]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 13:32:54.922299442 (574h3m54.793382477s) [upload=503272]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 13:32:55.174608877 (574h3m55.045691922s) [seg_pos={0 0}, upload=503272]: Begin upload segment...
Dec 29 13:32:55.174660355 (574h3m55.04574339s) [seg_pos={0 0}, upload=503272]: Done begin upload segment.
Dec 29 13:32:55.174695481 (574h3m55.045778537s) [noise=true , address=145.79.227.50:4374   , piece_id=5TLC6HM2UOM3KTY6BN7OGOADW45NHWGUDMJRWOOBDUN32UKJVSFQ, seg_pos={0 0}, upload=503272]: Uploading piece...
Dec 29 13:32:55.175815113 (574h3m55.046898148s) [noise=true , address=145.79.227.6:4257    , piece_id=QIQACXD6HMGZ2KUKJDBAYEWAGEED4SQVAJCPRUAEUFGPZ4QH7NAQ, seg_pos={0 0}, upload=503272]: Uploading piece...
Dec 29 13:32:55.176213182 (574h3m55.047296217s) [noise=true , address=145.79.227.11:4730   , piece_id=4LB3Q2UJFTMRAT6Z5574EBL624O4MIRKWBBUFOWPWYOFT3G6T46Q, seg_pos={0 0}, upload=503272]: Uploading piece...
Dec 29 13:32:55.176588396 (574h3m55.047671431s) [noise=true , address=45.91.116.11:4118    , piece_id=YBOVBJEKYNCV7PRTQJFGTCIQIC5FVZZ3YAV6GWZ3FAFEPMFCG5UA, seg_pos={0 0}, upload=503272]: Uploading piece...
Dec 29 13:32:55.176979351 (574h3m55.048062396s) [noise=true , address=145.79.227.14:4840   , piece_id=HXZYVRUV47CKYPJM74S7RW2FMPYEEA5NQHAJCWZDYSSZJYPX5BNQ, seg_pos={0 0}, upload=503272]: Uploading piece...
Dec 29 13:32:55.177410040 (574h3m55.048493076s) [noise=true , address=145.79.227.48:4302   , piece_id=6MXFBOKT5TLMHYWMOERJLRGYUQBURWVB5F7AHWANKWZ5QZDARUBQ, seg_pos={0 0}, upload=503272]: Uploading piece...
Dec 29 13:32:55.177831483 (574h3m55.048914528s) [noise=true , address=145.79.227.51:4409   , piece_id=LU2CIBANYYQCQXENEKWSHLARYP7DFXL2XIJFBND2SVVQ32Y2KXCA, seg_pos={0 0}, upload=503272]: Uploading piece...
Dec 29 13:32:55.178331434 (574h3m55.04941447s) [noise=true , address=77.90.31.14:4662     , piece_id=FWKIYJEBQOH5FPCVDEIXY2WCHTGMYWIWVTJRL7NFB6H3NHHGXSQA, seg_pos={0 0}, upload=503272]: Uploading piece...
Dec 29 13:32:55.180776510 (574h3m55.051859555s) [noise=true , address=145.79.227.45:4230   , piece_id=E4AXOSBNSI5ZJS27432DV7TA5A2WPMVUO7IDEVURTCHV4XJDPMSQ, seg_pos={0 0}, upload=503272]: Uploading piece...
Dec 29 13:32:55.182467270 (574h3m55.053550315s) [noise=true , address=145.79.227.26:4654   , piece_id=UPEP3SXGU4NA34GB22DDZR3LEQLERZEQRA5GOFWY7K3PP2UPHZMA, seg_pos={0 0}, upload=503272]: Uploading piece...
Dec 29 13:32:55.184402084 (574h3m55.055485119s) [noise=true , address=145.79.227.15:4876   , piece_id=OPTAI75X3O6GHARJSQYOYRC5WEMO6AJY2KITTZB76XXTR3QM7OMQ, seg_pos={0 0}, upload=503272]: Uploading piece...
Dec 29 13:32:55.185968987 (574h3m55.057052022s) [noise=true , address=45.91.116.4:4937     , piece_id=OSOFRPG5ZZJYAXCXYV6TXFRODS4A5JSPNJSSUKPKB26SETZUB4QA, seg_pos={0 0}, upload=503272]: Uploading piece...
Dec 29 13:32:55.186948073 (574h3m55.058031118s) [noise=true , address=45.91.116.6:4973     , piece_id=GKDOJZSB7BSLWRCXRUJAEUFHTKW3OSXIL2TRM4WTV6F5XD66Q2MQ, seg_pos={0 0}, upload=503272]: Uploading piece...
Dec 29 13:32:55.188248058 (574h3m55.059331103s) [noise=true , address=45.91.116.9:4261     , piece_id=MLZ6HV66PLZEHMANEUDUS7QKK5CQDYYVXUYGKKUIHOEVZD75V5FA, seg_pos={0 0}, upload=503272]: Uploading piece...
Dec 29 13:32:55.189439857 (574h3m55.060522892s) [noise=true , address=145.79.227.12:4767   , piece_id=YQZOLUCF3UZKVQ5FPCPY4CMKP5R7VEARS4ECNDJCCFLGKYIWMCEQ, seg_pos={0 0}, upload=503272]: Uploading piece...
Dec 29 13:32:55.190011426 (574h3m55.061094471s) [noise=true , address=145.79.227.39:4014   , piece_id=ADJG46R5VUJKFL7XCNDWWZ4572ZRSCIKYPO6MPPMCX5ZEURJTJBA, seg_pos={0 0}, upload=503272]: Uploading piece...
Dec 29 13:32:56.885830527 (574h3m56.756913562s) [upload=503272]: Next returned nil segment
Dec 29 13:32:56.885857578 (574h3m56.756940613s) [upload=503272]: Waiting for error group managing segments...
Dec 29 13:32:57.170240776 (574h3m57.041323811s) [noise=true , address=145.79.227.26:4654   , piece_id=UPEP3SXGU4NA34GB22DDZR3LEQLERZEQRA5GOFWY7K3PP2UPHZMA, seg_pos={0 0}, upload=503272]: Done uploading piece. err: <nil>
Dec 29 13:32:57.171808150 (574h3m57.042891185s) [noise=true , address=145.79.227.12:4767   , piece_id=YQZOLUCF3UZKVQ5FPCPY4CMKP5R7VEARS4ECNDJCCFLGKYIWMCEQ, seg_pos={0 0}, upload=503272]: Done uploading piece. err: <nil>
Dec 29 13:32:57.172047336 (574h3m57.043130371s) [noise=true , address=145.79.227.48:4302   , piece_id=6MXFBOKT5TLMHYWMOERJLRGYUQBURWVB5F7AHWANKWZ5QZDARUBQ, seg_pos={0 0}, upload=503272]: Done uploading piece. err: <nil>
Dec 29 13:32:57.174247305 (574h3m57.04533035s) [noise=true , address=145.79.227.45:4230   , piece_id=E4AXOSBNSI5ZJS27432DV7TA5A2WPMVUO7IDEVURTCHV4XJDPMSQ, seg_pos={0 0}, upload=503272]: Done uploading piece. err: <nil>
Dec 29 13:32:57.174831047 (574h3m57.045914082s) [noise=true , address=145.79.227.39:4014   , piece_id=ADJG46R5VUJKFL7XCNDWWZ4572ZRSCIKYPO6MPPMCX5ZEURJTJBA, seg_pos={0 0}, upload=503272]: Done uploading piece. err: <nil>
Dec 29 13:32:57.185543555 (574h3m57.05662659s) [noise=true , address=145.79.227.15:4876   , piece_id=OPTAI75X3O6GHARJSQYOYRC5WEMO6AJY2KITTZB76XXTR3QM7OMQ, seg_pos={0 0}, upload=503272]: Done uploading piece. err: <nil>
Dec 29 13:32:57.186233649 (574h3m57.057316685s) [noise=true , address=145.79.227.51:4409   , piece_id=LU2CIBANYYQCQXENEKWSHLARYP7DFXL2XIJFBND2SVVQ32Y2KXCA, seg_pos={0 0}, upload=503272]: Done uploading piece. err: <nil>
Dec 29 13:32:57.187737343 (574h3m57.058820378s) [noise=true , address=145.79.227.14:4840   , piece_id=HXZYVRUV47CKYPJM74S7RW2FMPYEEA5NQHAJCWZDYSSZJYPX5BNQ, seg_pos={0 0}, upload=503272]: Done uploading piece. err: <nil>
Dec 29 13:32:57.190448093 (574h3m57.061531139s) [noise=true , address=145.79.227.50:4374   , piece_id=5TLC6HM2UOM3KTY6BN7OGOADW45NHWGUDMJRWOOBDUN32UKJVSFQ, seg_pos={0 0}, upload=503272]: Done uploading piece. err: <nil>
Dec 29 13:32:57.195154666 (574h3m57.066237701s) [noise=true , address=145.79.227.11:4730   , piece_id=4LB3Q2UJFTMRAT6Z5574EBL624O4MIRKWBBUFOWPWYOFT3G6T46Q, seg_pos={0 0}, upload=503272]: Done uploading piece. err: <nil>
Dec 29 13:32:57.206946800 (574h3m57.078029845s) [noise=true , address=145.79.227.6:4257    , piece_id=QIQACXD6HMGZ2KUKJDBAYEWAGEED4SQVAJCPRUAEUFGPZ4QH7NAQ, seg_pos={0 0}, upload=503272]: Done uploading piece. err: <nil>
Dec 29 13:32:57.455818900 (574h3m57.326901935s) [noise=true , address=45.91.116.9:4261     , piece_id=MLZ6HV66PLZEHMANEUDUS7QKK5CQDYYVXUYGKKUIHOEVZD75V5FA, seg_pos={0 0}, upload=503272]: Done uploading piece. err: <nil>
Dec 29 13:32:57.487347825 (574h3m57.35843086s) [noise=true , address=45.91.116.6:4973     , piece_id=GKDOJZSB7BSLWRCXRUJAEUFHTKW3OSXIL2TRM4WTV6F5XD66Q2MQ, seg_pos={0 0}, upload=503272]: Done uploading piece. err: <nil>
Dec 29 13:32:57.487367633 (574h3m57.358450678s) [seg_pos={0 0}, upload=503272]: Segment reached optimal threshold of 13 pieces.
Dec 29 13:32:57.487547405 (574h3m57.35863044s) [noise=true , address=45.91.116.4:4937     , piece_id=OSOFRPG5ZZJYAXCXYV6TXFRODS4A5JSPNJSSUKPKB26SETZUB4QA, seg_pos={0 0}, upload=503272]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF
Dec 29 13:32:57.487556011 (574h3m57.358639046s) [noise=true , address=45.91.116.4:4937     , piece_id=OSOFRPG5ZZJYAXCXYV6TXFRODS4A5JSPNJSSUKPKB26SETZUB4QA, seg_pos={0 0}, upload=503272]: Long tail cancellation due to optimal threshold
Dec 29 13:32:57.487705877 (574h3m57.358788912s) [noise=true , address=45.91.116.11:4118    , piece_id=YBOVBJEKYNCV7PRTQJFGTCIQIC5FVZZ3YAV6GWZ3FAFEPMFCG5UA, seg_pos={0 0}, upload=503272]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF
Dec 29 13:32:57.487720986 (574h3m57.358804031s) [noise=true , address=45.91.116.11:4118    , piece_id=YBOVBJEKYNCV7PRTQJFGTCIQIC5FVZZ3YAV6GWZ3FAFEPMFCG5UA, seg_pos={0 0}, upload=503272]: Long tail cancellation due to optimal threshold
Dec 29 13:32:57.487799986 (574h3m57.358883021s) [noise=true , address=77.90.31.14:4662     , piece_id=FWKIYJEBQOH5FPCVDEIXY2WCHTGMYWIWVTJRL7NFB6H3NHHGXSQA, seg_pos={0 0}, upload=503272]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF
Dec 29 13:32:57.487807852 (574h3m57.358890887s) [noise=true , address=77.90.31.14:4662     , piece_id=FWKIYJEBQOH5FPCVDEIXY2WCHTGMYWIWVTJRL7NFB6H3NHHGXSQA, seg_pos={0 0}, upload=503272]: Long tail cancellation due to optimal threshold
Dec 29 13:32:57.487829302 (574h3m57.358912337s) [seg_pos={0 0}, upload=503272]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 13:32:57.487938229 (574h3m57.359021264s) [upload=503272]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 13:32:57.487946756 (574h3m57.359029801s) [upload=503272]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 13:32:57.778246089 (574h3m57.649329154s) [upload=503272]: Done uploading segments...
Dec 29 13:32:57.778276186 (574h3m57.649359221s) [upload=503272]: Upload finished. err: <nil>
Duration: 6.605313819s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
29 out of 60 is failed (threshold=3) [.XX...X.X.X.XXX.X.X.X.XX..X.XXX.XX..X.XX..XXX...X..X...X...X]
duration: 
30.003507616s
trace: 
https://jaeger.dev.storj.io/trace/4b55f632d9c143f
FAIL (2025-12-29 13:33:24 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: GKDOJZSB7BSLWRCXRUJAEUFHTKW3OSXIL2TRM4WTV6F5XD66Q2MQ) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: MLZ6HV66PLZEHMANEUDUS7QKK5CQDYYVXUYGKKUIHOEVZD75V5FA) context deadline exceeded
======LOGS======
Dec 29 13:32:54.921929839 (574h3m54.793012874s) [upload=503272]: Starting upload
Dec 29 13:32:54.921991305 (574h3m54.79307434s) [upload=503272]: Done starting upload
Dec 29 13:32:54.922216405 (574h3m54.79329944s) [upload=503272]: Uploading segments...
Dec 29 13:32:54.922257413 (574h3m54.793340458s) [upload=503272]: Got next segment. Inline: false
Dec 29 13:32:54.922295084 (574h3m54.793378119s) [upload=503272]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 13:32:54.922299442 (574h3m54.793382477s) [upload=503272]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 13:32:55.174608877 (574h3m55.045691922s) [seg_pos={0 0}, upload=503272]: Begin upload segment...
Dec 29 13:32:55.174660355 (574h3m55.04574339s) [seg_pos={0 0}, upload=503272]: Done begin upload segment.
Dec 29 13:32:55.174695481 (574h3m55.045778537s) [noise=true , address=145.79.227.50:4374   , piece_id=5TLC6HM2UOM3KTY6BN7OGOADW45NHWGUDMJRWOOBDUN32UKJVSFQ, seg_pos={0 0}, upload=503272]: Uploading piece...
Dec 29 13:32:55.175815113 (574h3m55.046898148s) [noise=true , address=145.79.227.6:4257    , piece_id=QIQACXD6HMGZ2KUKJDBAYEWAGEED4SQVAJCPRUAEUFGPZ4QH7NAQ, seg_pos={0 0}, upload=503272]: Uploading piece...
Dec 29 13:32:55.176213182 (574h3m55.047296217s) [noise=true , address=145.79.227.11:4730   , piece_id=4LB3Q2UJFTMRAT6Z5574EBL624O4MIRKWBBUFOWPWYOFT3G6T46Q, seg_pos={0 0}, upload=503272]: Uploading piece...
Dec 29 13:32:55.176588396 (574h3m55.047671431s) [noise=true , address=45.91.116.11:4118    , piece_id=YBOVBJEKYNCV7PRTQJFGTCIQIC5FVZZ3YAV6GWZ3FAFEPMFCG5UA, seg_pos={0 0}, upload=503272]: Uploading piece...
Dec 29 13:32:55.176979351 (574h3m55.048062396s) [noise=true , address=145.79.227.14:4840   , piece_id=HXZYVRUV47CKYPJM74S7RW2FMPYEEA5NQHAJCWZDYSSZJYPX5BNQ, seg_pos={0 0}, upload=503272]: Uploading piece...
Dec 29 13:32:55.177410040 (574h3m55.048493076s) [noise=true , address=145.79.227.48:4302   , piece_id=6MXFBOKT5TLMHYWMOERJLRGYUQBURWVB5F7AHWANKWZ5QZDARUBQ, seg_pos={0 0}, upload=503272]: Uploading piece...
Dec 29 13:32:55.177831483 (574h3m55.048914528s) [noise=true , address=145.79.227.51:4409   , piece_id=LU2CIBANYYQCQXENEKWSHLARYP7DFXL2XIJFBND2SVVQ32Y2KXCA, seg_pos={0 0}, upload=503272]: Uploading piece...
Dec 29 13:32:55.178331434 (574h3m55.04941447s) [noise=true , address=77.90.31.14:4662     , piece_id=FWKIYJEBQOH5FPCVDEIXY2WCHTGMYWIWVTJRL7NFB6H3NHHGXSQA, seg_pos={0 0}, upload=503272]: Uploading piece...
Dec 29 13:32:55.180776510 (574h3m55.051859555s) [noise=true , address=145.79.227.45:4230   , piece_id=E4AXOSBNSI5ZJS27432DV7TA5A2WPMVUO7IDEVURTCHV4XJDPMSQ, seg_pos={0 0}, upload=503272]: Uploading piece...
Dec 29 13:32:55.182467270 (574h3m55.053550315s) [noise=true , address=145.79.227.26:4654   , piece_id=UPEP3SXGU4NA34GB22DDZR3LEQLERZEQRA5GOFWY7K3PP2UPHZMA, seg_pos={0 0}, upload=503272]: Uploading piece...
Dec 29 13:32:55.184402084 (574h3m55.055485119s) [noise=true , address=145.79.227.15:4876   , piece_id=OPTAI75X3O6GHARJSQYOYRC5WEMO6AJY2KITTZB76XXTR3QM7OMQ, seg_pos={0 0}, upload=503272]: Uploading piece...
Dec 29 13:32:55.185968987 (574h3m55.057052022s) [noise=true , address=45.91.116.4:4937     , piece_id=OSOFRPG5ZZJYAXCXYV6TXFRODS4A5JSPNJSSUKPKB26SETZUB4QA, seg_pos={0 0}, upload=503272]: Uploading piece...
Dec 29 13:32:55.186948073 (574h3m55.058031118s) [noise=true , address=45.91.116.6:4973     , piece_id=GKDOJZSB7BSLWRCXRUJAEUFHTKW3OSXIL2TRM4WTV6F5XD66Q2MQ, seg_pos={0 0}, upload=503272]: Uploading piece...
Dec 29 13:32:55.188248058 (574h3m55.059331103s) [noise=true , address=45.91.116.9:4261     , piece_id=MLZ6HV66PLZEHMANEUDUS7QKK5CQDYYVXUYGKKUIHOEVZD75V5FA, seg_pos={0 0}, upload=503272]: Uploading piece...
Dec 29 13:32:55.189439857 (574h3m55.060522892s) [noise=true , address=145.79.227.12:4767   , piece_id=YQZOLUCF3UZKVQ5FPCPY4CMKP5R7VEARS4ECNDJCCFLGKYIWMCEQ, seg_pos={0 0}, upload=503272]: Uploading piece...
Dec 29 13:32:55.190011426 (574h3m55.061094471s) [noise=true , address=145.79.227.39:4014   , piece_id=ADJG46R5VUJKFL7XCNDWWZ4572ZRSCIKYPO6MPPMCX5ZEURJTJBA, seg_pos={0 0}, upload=503272]: Uploading piece...
Dec 29 13:32:56.885830527 (574h3m56.756913562s) [upload=503272]: Next returned nil segment
Dec 29 13:32:56.885857578 (574h3m56.756940613s) [upload=503272]: Waiting for error group managing segments...
Dec 29 13:32:57.170240776 (574h3m57.041323811s) [noise=true , address=145.79.227.26:4654   , piece_id=UPEP3SXGU4NA34GB22DDZR3LEQLERZEQRA5GOFWY7K3PP2UPHZMA, seg_pos={0 0}, upload=503272]: Done uploading piece. err: <nil>
Dec 29 13:32:57.171808150 (574h3m57.042891185s) [noise=true , address=145.79.227.12:4767   , piece_id=YQZOLUCF3UZKVQ5FPCPY4CMKP5R7VEARS4ECNDJCCFLGKYIWMCEQ, seg_pos={0 0}, upload=503272]: Done uploading piece. err: <nil>
Dec 29 13:32:57.172047336 (574h3m57.043130371s) [noise=true , address=145.79.227.48:4302   , piece_id=6MXFBOKT5TLMHYWMOERJLRGYUQBURWVB5F7AHWANKWZ5QZDARUBQ, seg_pos={0 0}, upload=503272]: Done uploading piece. err: <nil>
Dec 29 13:32:57.174247305 (574h3m57.04533035s) [noise=true , address=145.79.227.45:4230   , piece_id=E4AXOSBNSI5ZJS27432DV7TA5A2WPMVUO7IDEVURTCHV4XJDPMSQ, seg_pos={0 0}, upload=503272]: Done uploading piece. err: <nil>
Dec 29 13:32:57.174831047 (574h3m57.045914082s) [noise=true , address=145.79.227.39:4014   , piece_id=ADJG46R5VUJKFL7XCNDWWZ4572ZRSCIKYPO6MPPMCX5ZEURJTJBA, seg_pos={0 0}, upload=503272]: Done uploading piece. err: <nil>
Dec 29 13:32:57.185543555 (574h3m57.05662659s) [noise=true , address=145.79.227.15:4876   , piece_id=OPTAI75X3O6GHARJSQYOYRC5WEMO6AJY2KITTZB76XXTR3QM7OMQ, seg_pos={0 0}, upload=503272]: Done uploading piece. err: <nil>
Dec 29 13:32:57.186233649 (574h3m57.057316685s) [noise=true , address=145.79.227.51:4409   , piece_id=LU2CIBANYYQCQXENEKWSHLARYP7DFXL2XIJFBND2SVVQ32Y2KXCA, seg_pos={0 0}, upload=503272]: Done uploading piece. err: <nil>
Dec 29 13:32:57.187737343 (574h3m57.058820378s) [noise=true , address=145.79.227.14:4840   , piece_id=HXZYVRUV47CKYPJM74S7RW2FMPYEEA5NQHAJCWZDYSSZJYPX5BNQ, seg_pos={0 0}, upload=503272]: Done uploading piece. err: <nil>
Dec 29 13:32:57.190448093 (574h3m57.061531139s) [noise=true , address=145.79.227.50:4374   , piece_id=5TLC6HM2UOM3KTY6BN7OGOADW45NHWGUDMJRWOOBDUN32UKJVSFQ, seg_pos={0 0}, upload=503272]: Done uploading piece. err: <nil>
Dec 29 13:32:57.195154666 (574h3m57.066237701s) [noise=true , address=145.79.227.11:4730   , piece_id=4LB3Q2UJFTMRAT6Z5574EBL624O4MIRKWBBUFOWPWYOFT3G6T46Q, seg_pos={0 0}, upload=503272]: Done uploading piece. err: <nil>
Dec 29 13:32:57.206946800 (574h3m57.078029845s) [noise=true , address=145.79.227.6:4257    , piece_id=QIQACXD6HMGZ2KUKJDBAYEWAGEED4SQVAJCPRUAEUFGPZ4QH7NAQ, seg_pos={0 0}, upload=503272]: Done uploading piece. err: <nil>
Dec 29 13:32:57.455818900 (574h3m57.326901935s) [noise=true , address=45.91.116.9:4261     , piece_id=MLZ6HV66PLZEHMANEUDUS7QKK5CQDYYVXUYGKKUIHOEVZD75V5FA, seg_pos={0 0}, upload=503272]: Done uploading piece. err: <nil>
Dec 29 13:32:57.487347825 (574h3m57.35843086s) [noise=true , address=45.91.116.6:4973     , piece_id=GKDOJZSB7BSLWRCXRUJAEUFHTKW3OSXIL2TRM4WTV6F5XD66Q2MQ, seg_pos={0 0}, upload=503272]: Done uploading piece. err: <nil>
Dec 29 13:32:57.487367633 (574h3m57.358450678s) [seg_pos={0 0}, upload=503272]: Segment reached optimal threshold of 13 pieces.
Dec 29 13:32:57.487547405 (574h3m57.35863044s) [noise=true , address=45.91.116.4:4937     , piece_id=OSOFRPG5ZZJYAXCXYV6TXFRODS4A5JSPNJSSUKPKB26SETZUB4QA, seg_pos={0 0}, upload=503272]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF
Dec 29 13:32:57.487556011 (574h3m57.358639046s) [noise=true , address=45.91.116.4:4937     , piece_id=OSOFRPG5ZZJYAXCXYV6TXFRODS4A5JSPNJSSUKPKB26SETZUB4QA, seg_pos={0 0}, upload=503272]: Long tail cancellation due to optimal threshold
Dec 29 13:32:57.487705877 (574h3m57.358788912s) [noise=true , address=45.91.116.11:4118    , piece_id=YBOVBJEKYNCV7PRTQJFGTCIQIC5FVZZ3YAV6GWZ3FAFEPMFCG5UA, seg_pos={0 0}, upload=503272]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF
Dec 29 13:32:57.487720986 (574h3m57.358804031s) [noise=true , address=45.91.116.11:4118    , piece_id=YBOVBJEKYNCV7PRTQJFGTCIQIC5FVZZ3YAV6GWZ3FAFEPMFCG5UA, seg_pos={0 0}, upload=503272]: Long tail cancellation due to optimal threshold
Dec 29 13:32:57.487799986 (574h3m57.358883021s) [noise=true , address=77.90.31.14:4662     , piece_id=FWKIYJEBQOH5FPCVDEIXY2WCHTGMYWIWVTJRL7NFB6H3NHHGXSQA, seg_pos={0 0}, upload=503272]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF
Dec 29 13:32:57.487807852 (574h3m57.358890887s) [noise=true , address=77.90.31.14:4662     , piece_id=FWKIYJEBQOH5FPCVDEIXY2WCHTGMYWIWVTJRL7NFB6H3NHHGXSQA, seg_pos={0 0}, upload=503272]: Long tail cancellation due to optimal threshold
Dec 29 13:32:57.487829302 (574h3m57.358912337s) [seg_pos={0 0}, upload=503272]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 13:32:57.487938229 (574h3m57.359021264s) [upload=503272]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 13:32:57.487946756 (574h3m57.359029801s) [upload=503272]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 13:32:57.778246089 (574h3m57.649329154s) [upload=503272]: Done uploading segments...
Dec 29 13:32:57.778276186 (574h3m57.649359221s) [upload=503272]: Upload finished. err: <nil>
Duration: 30.003534989s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
5 out of 60 is failed (threshold=3) [................X....X.X....X.X.............................]
duration: 
19.630740464s
trace: 
https://jaeger.dev.storj.io/trace/312ddb07b2dee82a
FAIL (2025-12-29 13:32:58 UTC) 
orbiter warning: context deadline exceeded
Duration: 19.630768657s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
5 out of 60 is failed (threshold=3) [.................X....X.X....X.X............................]
duration: 
17.399731743s
trace: 
https://jaeger.dev.storj.io/trace/312ddb07b2dee82a
FAIL (2025-12-29 13:32:32 UTC) 
orbiter warning: context deadline exceeded
Duration: 17.399760989s edge gateway us-select-1