Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 356336 reports (Page 220 of 7127)
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)
antiblip: 
1 out of 60 is failed (threshold=3) [...........................................X................]
duration: 
6.429843742s
trace: 
https://jaeger.dev.storj.io/trace/414189176d03adca
FAIL (2025-12-30 14:50:54 UTC) 
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 6.429869871s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
47 out of 60 is failed (threshold=6) [XXXXXX.XXXXXXXXXXXXXXX.X..XX.XXXX.XXXXXXX.X..XX.X...XXXXXXXX]
duration: 
5.001437295s
trace: 
https://jaeger.dev.storj.io/trace/2b111f3e5f14ccf0
FAIL (2025-12-30 14:50:53 UTC) 
orbiter warning: context deadline exceeded
Duration: 5.001461601s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)
antiblip: 
2 out of 60 is failed (threshold=3) [.X........................................X.................]
duration: 
20.63834952s
trace: 
https://jaeger.dev.storj.io/trace/71938bc9cbb32ff3
FAIL (2025-12-30 14:50:52 UTC) 
orbiter warning: context deadline exceeded
Duration: 20.638363206s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
17 out of 60 is failed (threshold=3) [.....X...X....X......X.......X..X....X..X...X..XXX..XX..XX.X]
duration: 
30.002150503s
trace: 
https://jaeger.dev.storj.io/trace/36a669a34e93c5c5
FAIL (2025-12-30 14:50:43 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
4:50:14.466680953 (599h21m14.337763988s) [upload=525329]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 14:50:14.732049163 (599h21m14.603132208s) [seg_pos={0 0}, upload=525329]: Begin upload segment...
Dec 30 14:50:14.732134065 (599h21m14.60321711s) [seg_pos={0 0}, upload=525329]: Done begin upload segment.
Dec 30 14:50:14.732181214 (599h21m14.60326426s) [noise=true , address=145.79.227.14:4840   , piece_id=PRKQRIKODZDXP3XM5DD2S56O4XNAQCLPUIAYGRWS76U3BGTD2XKQ, seg_pos={0 0}, upload=525329]: Uploading piece...
Dec 30 14:50:14.732232512 (599h21m14.603315547s) [noise=true , address=145.79.227.19:4078   , piece_id=FQAHSLIAWW6MLFUU4QADVGVWKWASRLSPEXZQA6N6KFQND26XXLNA, seg_pos={0 0}, upload=525329]: Uploading piece...
Dec 30 14:50:14.732764315 (599h21m14.60384736s) [noise=true , address=45.91.116.10:4081    , piece_id=BDBAT6P4O7CEEDP72VZ2X5RMWDDDPRFVKMWO7WRLHDS62E6PCTSA, seg_pos={0 0}, upload=525329]: Uploading piece...
Dec 30 14:50:14.733226585 (599h21m14.60430963s) [noise=true , address=145.79.227.16:4911   , piece_id=SMWTADO2S3BI3H3KXWGLFMELRMO2SRHZO7S65D2CPPW4TN73BOGA, seg_pos={0 0}, upload=525329]: Uploading piece...
Dec 30 14:50:14.733398603 (599h21m14.604481638s) [noise=true , address=145.79.227.15:4876   , piece_id=DJTCCUZ4FMKXMJSEKBJDO5EVHPIR3OLT4AU3N2GNZRCTSQKZVOKA, seg_pos={0 0}, upload=525329]: Uploading piece...
Dec 30 14:50:14.733771303 (599h21m14.604854338s) [noise=true , address=45.91.116.11:4120    , piece_id=S4SL2DZYVEVTU7HMQLWK6WMRL3UJ23OELYSI57UFKRYXLBA2IXFA, seg_pos={0 0}, upload=525329]: Uploading piece...
Dec 30 14:50:14.733821107 (599h21m14.604904152s) [noise=true , address=145.79.227.18:4007   , piece_id=RMXJ7QGZD4EPNSVVK6COE4KD7FFWXNILU6YSMVUSYBFFIDU7QJJA, seg_pos={0 0}, upload=525329]: Uploading piece...
Dec 30 14:50:14.733880210 (599h21m14.604963255s) [noise=true , address=45.91.116.7:4009     , piece_id=REKSBZJCHDSQX3LIFTMKUQ24FU6F4M6GIJQCBBR2U525NJYE2TLA, seg_pos={0 0}, upload=525329]: Uploading piece...
Dec 30 14:50:14.734225738 (599h21m14.605308773s) [noise=true , address=145.79.227.9:4365    , piece_id=P4J5G3C5SD6I5PRNHAYYCORRHFIKFEOEKEP47J6HCN3XAT2ZDYHA, seg_pos={0 0}, upload=525329]: Uploading piece...
Dec 30 14:50:14.734350395 (599h21m14.60543344s) [noise=true , address=45.91.116.4:4937     , piece_id=PIGD3Z4YPNSXQNDKVNCKEJENLCFJ7VJHZNZDAQN2X7NEHLYUVNYA, seg_pos={0 0}, upload=525329]: Uploading piece...
Dec 30 14:50:14.734722013 (599h21m14.605805048s) [noise=true , address=145.79.227.38:4977   , piece_id=MZDAQSTHAOU4EXRUD4ICIAYRG2UT76PMHS5KFDNJ4UY3PWDAVTFA, seg_pos={0 0}, upload=525329]: Uploading piece...
Dec 30 14:50:14.735274685 (599h21m14.60635772s) [noise=true , address=145.79.227.39:4013   , piece_id=G2YNEAUK7LP7M2STK2I3F5RRYNKVAZVPXZDTDWT2FI2WZQAN5AEA, seg_pos={0 0}, upload=525329]: Uploading piece...
Dec 30 14:50:14.735750731 (599h21m14.606833806s) [noise=true , address=145.79.227.52:4447   , piece_id=SQEE2DULLMGRG6PQKXS7QIHMPOK5VJENOI7NZDB7YTD6573PQ3TQ, seg_pos={0 0}, upload=525329]: Uploading piece...
Dec 30 14:50:14.736295158 (599h21m14.607378203s) [noise=true , address=45.91.116.9:4261     , piece_id=34PZQDMKMLVFTL3ILQZREOKBJQ3NCJNSJIDWEXSZANUZ4LQBE7DA, seg_pos={0 0}, upload=525329]: Uploading piece...
Dec 30 14:50:14.740767134 (599h21m14.611850209s) [noise=true , address=77.90.31.11:4556     , piece_id=SM36SIBJBQ5A7PWBRWHQ3R6STOOEAW6VOEATFVG3NTXGHCOKZPZA, seg_pos={0 0}, upload=525329]: Uploading piece...
Dec 30 14:50:14.742815944 (599h21m14.613898999s) [noise=true , address=145.79.227.32:4834   , piece_id=S7AAJEHAGNEV4GKZQU5GOFEULISNBOWTZFGVEOCTH7LHP3HYOGPA, seg_pos={0 0}, upload=525329]: Uploading piece...
Dec 30 14:50:16.444089160 (599h21m16.315172195s) [upload=525329]: Next returned nil segment
Dec 30 14:50:16.444101634 (599h21m16.315184669s) [upload=525329]: Waiting for error group managing segments...
Dec 30 14:50:16.730994049 (599h21m16.602077094s) [noise=true , address=145.79.227.9:4365    , piece_id=P4J5G3C5SD6I5PRNHAYYCORRHFIKFEOEKEP47J6HCN3XAT2ZDYHA, seg_pos={0 0}, upload=525329]: Done uploading piece. err: <nil>
Dec 30 14:50:16.734857195 (599h21m16.60594023s) [noise=true , address=145.79.227.19:4078   , piece_id=FQAHSLIAWW6MLFUU4QADVGVWKWASRLSPEXZQA6N6KFQND26XXLNA, seg_pos={0 0}, upload=525329]: Done uploading piece. err: <nil>
Dec 30 14:50:16.743018857 (599h21m16.614101902s) [noise=true , address=145.79.227.18:4007   , piece_id=RMXJ7QGZD4EPNSVVK6COE4KD7FFWXNILU6YSMVUSYBFFIDU7QJJA, seg_pos={0 0}, upload=525329]: Done uploading piece. err: <nil>
Dec 30 14:50:16.746153926 (599h21m16.617236971s) [noise=true , address=145.79.227.14:4840   , piece_id=PRKQRIKODZDXP3XM5DD2S56O4XNAQCLPUIAYGRWS76U3BGTD2XKQ, seg_pos={0 0}, upload=525329]: Done uploading piece. err: <nil>
Dec 30 14:50:16.747366294 (599h21m16.618449339s) [noise=true , address=145.79.227.52:4447   , piece_id=SQEE2DULLMGRG6PQKXS7QIHMPOK5VJENOI7NZDB7YTD6573PQ3TQ, seg_pos={0 0}, upload=525329]: Done uploading piece. err: <nil>
Dec 30 14:50:16.755473862 (599h21m16.626556907s) [noise=true , address=145.79.227.38:4977   , piece_id=MZDAQSTHAOU4EXRUD4ICIAYRG2UT76PMHS5KFDNJ4UY3PWDAVTFA, seg_pos={0 0}, upload=525329]: Done uploading piece. err: <nil>
Dec 30 14:50:16.767591957 (599h21m16.638675002s) [noise=true , address=145.79.227.32:4834   , piece_id=S7AAJEHAGNEV4GKZQU5GOFEULISNBOWTZFGVEOCTH7LHP3HYOGPA, seg_pos={0 0}, upload=525329]: Done uploading piece. err: <nil>
Dec 30 14:50:16.798459664 (599h21m16.669542699s) [noise=true , address=145.79.227.16:4911   , piece_id=SMWTADO2S3BI3H3KXWGLFMELRMO2SRHZO7S65D2CPPW4TN73BOGA, seg_pos={0 0}, upload=525329]: Done uploading piece. err: <nil>
Dec 30 14:50:16.815726692 (599h21m16.686809747s) [noise=true , address=145.79.227.39:4013   , piece_id=G2YNEAUK7LP7M2STK2I3F5RRYNKVAZVPXZDTDWT2FI2WZQAN5AEA, seg_pos={0 0}, upload=525329]: Done uploading piece. err: <nil>
Dec 30 14:50:16.822440136 (599h21m16.693523181s) [noise=true , address=145.79.227.15:4876   , piece_id=DJTCCUZ4FMKXMJSEKBJDO5EVHPIR3OLT4AU3N2GNZRCTSQKZVOKA, seg_pos={0 0}, upload=525329]: Done uploading piece. err: <nil>
Dec 30 14:50:16.990580600 (599h21m16.861663645s) [noise=true , address=77.90.31.11:4556     , piece_id=SM36SIBJBQ5A7PWBRWHQ3R6STOOEAW6VOEATFVG3NTXGHCOKZPZA, seg_pos={0 0}, upload=525329]: Done uploading piece. err: <nil>
Dec 30 14:50:17.443121044 (599h21m17.31420408s) [noise=true , address=45.91.116.10:4081    , piece_id=BDBAT6P4O7CEEDP72VZ2X5RMWDDDPRFVKMWO7WRLHDS62E6PCTSA, seg_pos={0 0}, upload=525329]: Done uploading piece. err: <nil>
Dec 30 14:50:17.538073478 (599h21m17.409156523s) [noise=true , address=45.91.116.11:4120    , piece_id=S4SL2DZYVEVTU7HMQLWK6WMRL3UJ23OELYSI57UFKRYXLBA2IXFA, seg_pos={0 0}, upload=525329]: Done uploading piece. err: <nil>
Dec 30 14:50:17.538093586 (599h21m17.409176621s) [seg_pos={0 0}, upload=525329]: Segment reached optimal threshold of 13 pieces.
Dec 30 14:50:17.538285481 (599h21m17.409368526s) [noise=true , address=45.91.116.9:4261     , piece_id=34PZQDMKMLVFTL3ILQZREOKBJQ3NCJNSJIDWEXSZANUZ4LQBE7DA, seg_pos={0 0}, upload=525329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:54128->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:54128->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54128->45.91.116.9:4261: use of closed network connection
Dec 30 14:50:17.538294368 (599h21m17.409377403s) [noise=true , address=45.91.116.9:4261     , piece_id=34PZQDMKMLVFTL3ILQZREOKBJQ3NCJNSJIDWEXSZANUZ4LQBE7DA, seg_pos={0 0}, upload=525329]: Long tail cancellation due to optimal threshold
Dec 30 14:50:17.538403125 (599h21m17.40948617s) [noise=true , address=45.91.116.7:4009     , piece_id=REKSBZJCHDSQX3LIFTMKUQ24FU6F4M6GIJQCBBR2U525NJYE2TLA, seg_pos={0 0}, upload=525329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:48588->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:48588->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48588->45.91.116.7:4009: use of closed network connection
Dec 30 14:50:17.538423344 (599h21m17.409506379s) [noise=true , address=45.91.116.7:4009     , piece_id=REKSBZJCHDSQX3LIFTMKUQ24FU6F4M6GIJQCBBR2U525NJYE2TLA, seg_pos={0 0}, upload=525329]: Long tail cancellation due to optimal threshold
Dec 30 14:50:17.538488668 (599h21m17.409571713s) [noise=true , address=45.91.116.4:4937     , piece_id=PIGD3Z4YPNSXQNDKVNCKEJENLCFJ7VJHZNZDAQN2X7NEHLYUVNYA, seg_pos={0 0}, upload=525329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:49716->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:49716->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49716->45.91.116.4:4937: use of closed network connection
Dec 30 14:50:17.538495491 (599h21m17.409578536s) [noise=true , address=45.91.116.4:4937     , piece_id=PIGD3Z4YPNSXQNDKVNCKEJENLCFJ7VJHZNZDAQN2X7NEHLYUVNYA, seg_pos={0 0}, upload=525329]: Long tail cancellation due to optimal threshold
Dec 30 14:50:17.538593408 (599h21m17.409676443s) [seg_pos={0 0}, upload=525329]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 14:50:17.538717454 (599h21m17.409800499s) [upload=525329]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 14:50:17.538724858 (599h21m17.409807893s) [upload=525329]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 14:50:17.873898759 (599h21m17.744981794s) [upload=525329]: Done uploading segments...
Dec 30 14:50:17.873918987 (599h21m17.745002032s) [upload=525329]: Upload finished. err: <nil>
Duration: 30.002178418s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)
antiblip: 
3 out of 60 is failed (threshold=3) [X.X........................................X................]
duration: 
32.120770869s
trace: 
https://jaeger.dev.storj.io/trace/71938bc9cbb32ff3
FAIL (2025-12-30 14:50:31 UTC) 
orbiter warning: context deadline exceeded
Duration: 32.120796627s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
14 out of 60 is failed (threshold=3) [XXX.....X......X......X........X.X.X..X..X.....XXX..........]
duration: 
25.071825872s
trace: 
https://jaeger.dev.storj.io/trace/41ecdbaa0176ef93
FAIL (2025-12-30 14:50:19 UTC) 
orbiter warning: context deadline exceeded
Duration: 25.071858675s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [.............................X...........................X..]
duration: 
30.629287589s
trace: 
https://jaeger.dev.storj.io/trace/328a692d1ab4d967
FAIL (2025-12-30 14:50:15 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 17ddUMj4ZejcjjWGzkGEM63vbCoEPnu7ePznzAPyzQofW8Q4gd, Piece ID: AP324FY2ZWKR4JNW7WET4XMLHNIR53YJHS4EYA7T2NFQMDAPJGOQ) context deadline exceeded; piecestore: (Node ID: 128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, Piece ID: SRPWR2M2GYYA3JOMGHMDZHHXI2FLDJCIQROQZAVFDTV6ASPDDVMA) context deadline exceeded; piecestore: (Node ID: 1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, Piece ID: 7RS2ACDDX55LLN75FUYF7YFC72ACCLFSKPGHBDAMC4K5RPZBEK2A) context deadline exceeded; piecestore: (Node ID: 12qXhyMbfXELW1Vg5z9NpqFX1fm3v2pGEaA56qSc8kHefPNNNYs, Piece ID: ZUQVNQBMWUN7JD5GNPBNIVDHHHDAJJH7DGWRJZ2GBYGQMF25ZBTQ) context deadline exceeded
======LOGS======
oise=true , address=145.79.227.43:4180   , piece_id=IQEZVMY7IRLS5VQSPUXNUZSP3XZZ67X4TDLDONUT4MVRNVNEAANQ, seg_pos={0 0}, upload=525299]: Uploading piece...
Dec 30 14:48:11.029459924 (599h19m10.900542959s) [noise=true , address=64.49.13.155:4920    , piece_id=D3YEBC7PPVGA5DCFWUWR6BFHV3N43WE4RPAS7LT7GX5XIRMLPEUQ, seg_pos={0 0}, upload=525299]: Uploading piece...
Dec 30 14:48:11.029880215 (599h19m10.90096326s) [noise=true , address=145.79.227.34:4935   , piece_id=C3EF7WUSWDMZKVMULEXZHMQOKRH2ODJVVHRKIV57GYH4INI5QIJA, seg_pos={0 0}, upload=525299]: Uploading piece...
Dec 30 14:48:11.030199834 (599h19m10.901282869s) [noise=true , address=145.79.227.49:4347   , piece_id=TILSKTAE3CDAL4UIYZK5POLHEAJCCI5PFJJX5SNEBYIJHP2TPFHQ, seg_pos={0 0}, upload=525299]: Uploading piece...
Dec 30 14:48:11.030300455 (599h19m10.90138349s) [noise=true , address=145.79.227.4:4210    , piece_id=KWBC7SM7JYHXLRPBOOXGU7QQKDJWRB2U3AOBDQCKLLTZZLLFXDMQ, seg_pos={0 0}, upload=525299]: Uploading piece...
Dec 30 14:48:11.030568916 (599h19m10.901651951s) [noise=true , address=77.90.31.14:4689     , piece_id=7RS2ACDDX55LLN75FUYF7YFC72ACCLFSKPGHBDAMC4K5RPZBEK2A, seg_pos={0 0}, upload=525299]: Uploading piece...
Dec 30 14:48:11.030753688 (599h19m10.901836724s) [noise=true , address=77.90.31.19:4860     , piece_id=AP324FY2ZWKR4JNW7WET4XMLHNIR53YJHS4EYA7T2NFQMDAPJGOQ, seg_pos={0 0}, upload=525299]: Uploading piece...
Dec 30 14:48:11.030910497 (599h19m10.901993532s) [noise=true , address=77.90.31.11:4586     , piece_id=XJDAQMNAIBFY6EICSLSIU52WIAIJKNPODKNYRWECS6CV7GELAMGQ, seg_pos={0 0}, upload=525299]: Uploading piece...
Dec 30 14:48:11.031175852 (599h19m10.902258887s) [noise=true , address=145.79.227.44:4212   , piece_id=QTY5RT75QRIUL7GBRB3V4LJK4JL4U4IN4RGZMBV7TOKZ75DOSQEQ, seg_pos={0 0}, upload=525299]: Uploading piece...
Dec 30 14:48:11.031220547 (599h19m10.902303592s) [noise=true , address=45.91.116.12:4252    , piece_id=ISEASO5OV7UBUPGR3B4CJJU3QCDYLC365PT4B4BKAIN3W7GTX4BQ, seg_pos={0 0}, upload=525299]: Uploading piece...
Dec 30 14:48:11.031518976 (599h19m10.902602011s) [noise=true , address=64.49.13.169:4709    , piece_id=PVOTCKOHNI7LLPOEZNQ7RHQADMYDGVVAD5HGLRGYDKAZ7W22VUFA, seg_pos={0 0}, upload=525299]: Uploading piece...
Dec 30 14:48:11.031659844 (599h19m10.902742879s) [noise=true , address=77.90.31.13:4633     , piece_id=ZUQVNQBMWUN7JD5GNPBNIVDHHHDAJJH7DGWRJZ2GBYGQMF25ZBTQ, seg_pos={0 0}, upload=525299]: Uploading piece...
Dec 30 14:48:11.031896624 (599h19m10.902979659s) [noise=true , address=77.90.31.18:4812     , piece_id=SRPWR2M2GYYA3JOMGHMDZHHXI2FLDJCIQROQZAVFDTV6ASPDDVMA, seg_pos={0 0}, upload=525299]: Uploading piece...
Dec 30 14:48:11.032080996 (599h19m10.903164031s) [noise=true , address=145.79.227.30:4769   , piece_id=WL2KVAFIDCIBEX6PGFVT5M4JAC525YC3MAU4SVVUMFS3FAB6QBNQ, seg_pos={0 0}, upload=525299]: Uploading piece...
Dec 30 14:48:11.032234387 (599h19m10.903317422s) [noise=true , address=145.79.227.35:4513   , piece_id=SABYA6SJYZ65IFUYFGSYU6UUS6QSXFCIQCT7BRSJM7BCWHQILYDA, seg_pos={0 0}, upload=525299]: Uploading piece...
Dec 30 14:48:11.032571650 (599h19m10.903654685s) [noise=true , address=64.49.13.137:4494    , piece_id=RRKIPPWSAWV7QILA4PGQJ2IKAGJ6BSMP7OPFRQJJK4UWWWQNXVZA, seg_pos={0 0}, upload=525299]: Uploading piece...
Dec 30 14:48:12.423070865 (599h19m12.2941539s) [upload=525299]: Next returned nil segment
Dec 30 14:48:12.423093118 (599h19m12.294176153s) [upload=525299]: Waiting for error group managing segments...
Dec 30 14:48:12.831938369 (599h19m12.703021404s) [noise=true , address=64.49.13.27:4965     , piece_id=XUTKVFCA5BJNZIBJCBWZ75K3B3TCWSZ5U5ESHATA6K376Z2YNXKA, seg_pos={0 0}, upload=525299]: Done uploading piece. err: <nil>
Dec 30 14:48:12.868364350 (599h19m12.739447395s) [noise=true , address=64.49.13.137:4494    , piece_id=RRKIPPWSAWV7QILA4PGQJ2IKAGJ6BSMP7OPFRQJJK4UWWWQNXVZA, seg_pos={0 0}, upload=525299]: Done uploading piece. err: <nil>
Dec 30 14:48:12.959516487 (599h19m12.830599532s) [noise=true , address=64.49.13.155:4920    , piece_id=D3YEBC7PPVGA5DCFWUWR6BFHV3N43WE4RPAS7LT7GX5XIRMLPEUQ, seg_pos={0 0}, upload=525299]: Done uploading piece. err: <nil>
Dec 30 14:48:12.975446991 (599h19m12.846530026s) [noise=true , address=145.79.227.4:4210    , piece_id=KWBC7SM7JYHXLRPBOOXGU7QQKDJWRB2U3AOBDQCKLLTZZLLFXDMQ, seg_pos={0 0}, upload=525299]: Done uploading piece. err: <nil>
Dec 30 14:48:13.079109379 (599h19m12.950192414s) [noise=true , address=77.90.31.19:4860     , piece_id=AP324FY2ZWKR4JNW7WET4XMLHNIR53YJHS4EYA7T2NFQMDAPJGOQ, seg_pos={0 0}, upload=525299]: Done uploading piece. err: <nil>
Dec 30 14:48:13.091745701 (599h19m12.962828746s) [noise=true , address=77.90.31.13:4633     , piece_id=ZUQVNQBMWUN7JD5GNPBNIVDHHHDAJJH7DGWRJZ2GBYGQMF25ZBTQ, seg_pos={0 0}, upload=525299]: Done uploading piece. err: <nil>
Dec 30 14:48:13.101549980 (599h19m12.972633025s) [noise=true , address=77.90.31.18:4812     , piece_id=SRPWR2M2GYYA3JOMGHMDZHHXI2FLDJCIQROQZAVFDTV6ASPDDVMA, seg_pos={0 0}, upload=525299]: Done uploading piece. err: <nil>
Dec 30 14:48:13.270615245 (599h19m13.14169828s) [noise=true , address=45.91.116.12:4252    , piece_id=ISEASO5OV7UBUPGR3B4CJJU3QCDYLC365PT4B4BKAIN3W7GTX4BQ, seg_pos={0 0}, upload=525299]: Done uploading piece. err: <nil>
Dec 30 14:48:13.396635416 (599h19m13.267718461s) [noise=true , address=77.90.31.11:4586     , piece_id=XJDAQMNAIBFY6EICSLSIU52WIAIJKNPODKNYRWECS6CV7GELAMGQ, seg_pos={0 0}, upload=525299]: Done uploading piece. err: <nil>
Dec 30 14:48:13.735330842 (599h19m13.606413877s) [noise=true , address=145.79.227.49:4347   , piece_id=TILSKTAE3CDAL4UIYZK5POLHEAJCCI5PFJJX5SNEBYIJHP2TPFHQ, seg_pos={0 0}, upload=525299]: Done uploading piece. err: <nil>
Dec 30 14:48:13.775816923 (599h19m13.646899958s) [noise=true , address=145.79.227.30:4769   , piece_id=WL2KVAFIDCIBEX6PGFVT5M4JAC525YC3MAU4SVVUMFS3FAB6QBNQ, seg_pos={0 0}, upload=525299]: Done uploading piece. err: <nil>
Dec 30 14:48:13.782446286 (599h19m13.653529321s) [noise=true , address=145.79.227.43:4180   , piece_id=IQEZVMY7IRLS5VQSPUXNUZSP3XZZ67X4TDLDONUT4MVRNVNEAANQ, seg_pos={0 0}, upload=525299]: Done uploading piece. err: <nil>
Dec 30 14:48:13.833466567 (599h19m13.704549602s) [noise=true , address=145.79.227.44:4212   , piece_id=QTY5RT75QRIUL7GBRB3V4LJK4JL4U4IN4RGZMBV7TOKZ75DOSQEQ, seg_pos={0 0}, upload=525299]: Done uploading piece. err: <nil>
Dec 30 14:48:13.934059641 (599h19m13.805142676s) [noise=true , address=77.90.31.14:4689     , piece_id=7RS2ACDDX55LLN75FUYF7YFC72ACCLFSKPGHBDAMC4K5RPZBEK2A, seg_pos={0 0}, upload=525299]: Done uploading piece. err: <nil>
Dec 30 14:48:13.934166574 (599h19m13.805249609s) [seg_pos={0 0}, upload=525299]: Segment reached optimal threshold of 14 pieces.
Dec 30 14:48:13.934461506 (599h19m13.805544551s) [noise=true , address=145.79.227.35:4513   , piece_id=SABYA6SJYZ65IFUYFGSYU6UUS6QSXFCIQCT7BRSJM7BCWHQILYDA, seg_pos={0 0}, upload=525299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pyyW8Gdqg6j9YGMtCUBhFnoNXryvPMUy19x815dp2HV3Xqr8h): protocol: expected piece hash; context canceled; EOF
Dec 30 14:48:13.934472587 (599h19m13.805555622s) [noise=true , address=145.79.227.35:4513   , piece_id=SABYA6SJYZ65IFUYFGSYU6UUS6QSXFCIQCT7BRSJM7BCWHQILYDA, seg_pos={0 0}, upload=525299]: Long tail cancellation due to optimal threshold
Dec 30 14:48:13.934771517 (599h19m13.805854552s) [noise=true , address=77.90.31.12:4620     , piece_id=TXXA7FP2AVAOXTQ7B4Y2XCCCDW6HBJQ3BY6EMOV6X4IHWNUHQ7UQ, seg_pos={0 0}, upload=525299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe): protocol: expected piece hash; context canceled; EOF
Dec 30 14:48:13.934779772 (599h19m13.805862807s) [noise=true , address=77.90.31.12:4620     , piece_id=TXXA7FP2AVAOXTQ7B4Y2XCCCDW6HBJQ3BY6EMOV6X4IHWNUHQ7UQ, seg_pos={0 0}, upload=525299]: Long tail cancellation due to optimal threshold
Dec 30 14:48:13.935018687 (599h19m13.806101732s) [noise=true , address=145.79.227.34:4935   , piece_id=C3EF7WUSWDMZKVMULEXZHMQOKRH2ODJVVHRKIV57GYH4INI5QIJA, seg_pos={0 0}, upload=525299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ks2kQWUmV377HgGbyoTt96CTyK1P1nH8QtZQzh3xvnFCYaD8X): protocol: expected piece hash; context canceled; EOF
Dec 30 14:48:13.935027324 (599h19m13.806110369s) [noise=true , address=145.79.227.34:4935   , piece_id=C3EF7WUSWDMZKVMULEXZHMQOKRH2ODJVVHRKIV57GYH4INI5QIJA, seg_pos={0 0}, upload=525299]: Long tail cancellation due to optimal threshold
Dec 30 14:48:13.935192078 (599h19m13.806275113s) [noise=true , address=64.49.13.169:4709    , piece_id=PVOTCKOHNI7LLPOEZNQ7RHQADMYDGVVAD5HGLRGYDKAZ7W22VUFA, seg_pos={0 0}, upload=525299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:55418->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:55418->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55418->64.49.13.169:4709: use of closed network connection
Dec 30 14:48:13.935200634 (599h19m13.806283669s) [noise=true , address=64.49.13.169:4709    , piece_id=PVOTCKOHNI7LLPOEZNQ7RHQADMYDGVVAD5HGLRGYDKAZ7W22VUFA, seg_pos={0 0}, upload=525299]: Long tail cancellation due to optimal threshold
Dec 30 14:48:13.935254516 (599h19m13.806337561s) [seg_pos={0 0}, upload=525299]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 14:48:13.935335981 (599h19m13.806419027s) [upload=525299]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 14:48:13.935342193 (599h19m13.806425238s) [upload=525299]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 14:48:14.224449907 (599h19m14.095532972s) [upload=525299]: Done uploading segments...
Dec 30 14:48:14.224476296 (599h19m14.095559332s) [upload=525299]: Upload finished. err: <nil>
Duration: 30.629306425s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
17 out of 60 is failed (threshold=3) [X.....X...X....X......X.......X..X....X..X...X..XXX..XX..XX.]
duration: 
5.943482296s
trace: 
https://jaeger.dev.storj.io/trace/688751829c2f4ed5
FAIL (2025-12-30 14:49:58 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: AN3KCDQVIR4XVSZ6DF6IAPBVKLXTWPLNNUMELRUNX7HHFPELYI7Q) context deadline exceeded; piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: BHUJU7RKWLWS63DGJDWTUFDG4DHNZGSLGJ7JXW6Q6TQ4BP33MXDQ) context deadline exceeded
======LOGS======
Dec 30 14:49:15.821133341 (599h20m15.692216386s) [upload=525314]: Starting upload
Dec 30 14:49:15.821213894 (599h20m15.692296939s) [upload=525314]: Done starting upload
Dec 30 14:49:15.821291121 (599h20m15.692374166s) [upload=525314]: Uploading segments...
Dec 30 14:49:15.821470293 (599h20m15.692553328s) [upload=525314]: Got next segment. Inline: false
Dec 30 14:49:15.821510500 (599h20m15.692593535s) [upload=525314]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 14:49:15.821515269 (599h20m15.692598304s) [upload=525314]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 14:49:16.064550864 (599h20m15.935633909s) [seg_pos={0 0}, upload=525314]: Begin upload segment...
Dec 30 14:49:16.064606450 (599h20m15.935689485s) [seg_pos={0 0}, upload=525314]: Done begin upload segment.
Dec 30 14:49:16.064632299 (599h20m15.935715335s) [noise=true , address=145.79.227.11:4732   , piece_id=BQCXKQ2LKP5SSWQPDZZFHG4YJWKVL7ZKYNCPVSV324M2WGPDSHIA, seg_pos={0 0}, upload=525314]: Uploading piece...
Dec 30 14:49:16.064681142 (599h20m15.935764177s) [noise=true , address=145.79.227.2:4115    , piece_id=CVRG3KYICA6HRB72G5ZFLYXAVU2VBE7LSHWDUPXJ5ZI2CEQ646JQ, seg_pos={0 0}, upload=525314]: Uploading piece...
Dec 30 14:49:16.064775853 (599h20m15.935858898s) [noise=true , address=145.79.227.32:4836   , piece_id=G5L7AXETN2N33UUWO7YBWFRO4HQLB6AI77C3FXAFRD45LOFHWYBA, seg_pos={0 0}, upload=525314]: Uploading piece...
Dec 30 14:49:16.065109658 (599h20m15.936192693s) [noise=true , address=145.79.227.9:4367    , piece_id=AHAOE6LGGXK76FXMHIEYQ5DFOI6Z2IB5PNSPVA53H25RPBJMDERQ, seg_pos={0 0}, upload=525314]: Uploading piece...
Dec 30 14:49:16.065931713 (599h20m15.937014749s) [noise=true , address=145.79.227.24:4581   , piece_id=7AJERH43PANL5BEBYVP2ZJ4LWGYYI7XD2UGPFMMQOSWHNOCSJZ5Q, seg_pos={0 0}, upload=525314]: Uploading piece...
Dec 30 14:49:16.066105454 (599h20m15.937188489s) [noise=true , address=145.79.227.8:4330    , piece_id=TFARMBVGBDUVJ33PNJW5OO27NW7T6COI55IA4U3I6QX6YW3QUOQA, seg_pos={0 0}, upload=525314]: Uploading piece...
Dec 30 14:49:16.066246322 (599h20m15.937329368s) [noise=true , address=77.90.31.19:4844     , piece_id=EVZOFDUMQLTTPS4X5IQEZTXE5L6GUNYXESGBGXLY3MJEACE4CAWA, seg_pos={0 0}, upload=525314]: Uploading piece...
Dec 30 14:49:16.066560911 (599h20m15.937643957s) [noise=true , address=45.91.116.6:4976     , piece_id=PJUSIJPXPT353JLGPDBPV3WWGLMI6S2UMXPEM43UUIERPYDXGWFA, seg_pos={0 0}, upload=525314]: Uploading piece...
Dec 30 14:49:16.067090730 (599h20m15.938173765s) [noise=true , address=77.90.31.21:4878     , piece_id=PGA6V26MV4NYZA6EWNUF2GE65UBK2ZAOG23T7B2UCEH7G2KC2E4Q, seg_pos={0 0}, upload=525314]: Uploading piece...
Dec 30 14:49:16.067698037 (599h20m15.938781082s) [noise=true , address=145.79.227.36:4519   , piece_id=YNDF6NPMJQ5VUYXVLWRIAD3SJLYOKKAGLC45WURNSBHZ3SLRE2AQ, seg_pos={0 0}, upload=525314]: Uploading piece...
Dec 30 14:49:16.068034968 (599h20m15.939118003s) [noise=true , address=77.90.31.10:4517     , piece_id=AN3KCDQVIR4XVSZ6DF6IAPBVKLXTWPLNNUMELRUNX7HHFPELYI7Q, seg_pos={0 0}, upload=525314]: Uploading piece...
Dec 30 14:49:16.068256460 (599h20m15.939339505s) [noise=true , address=145.79.227.5:4223    , piece_id=UABOB6ROSJJJJSUVZ3BSFLSHBTIAMDK2W2JN5EIXDTO2FEKWCC7Q, seg_pos={0 0}, upload=525314]: Uploading piece...
Dec 30 14:49:16.068670930 (599h20m15.939753965s) [noise=true , address=45.91.116.7:4012     , piece_id=V5GP7DRHXY63ADQLGTKEIDC7OGGR6FVLOGLAZV4FEF65PKVYP5UQ, seg_pos={0 0}, upload=525314]: Uploading piece...
Dec 30 14:49:16.068792280 (599h20m15.939875325s) [noise=true , address=77.90.31.12:4590     , piece_id=BHUJU7RKWLWS63DGJDWTUFDG4DHNZGSLGJ7JXW6Q6TQ4BP33MXDQ, seg_pos={0 0}, upload=525314]: Uploading piece...
Dec 30 14:49:16.069004735 (599h20m15.94008778s) [noise=true , address=145.79.227.50:4375   , piece_id=2QOU365LVL3EOSNAB55FFBLUMH7W7F6BXV4ZY4ICSMMFO626CZGQ, seg_pos={0 0}, upload=525314]: Uploading piece...
Dec 30 14:49:16.069319354 (599h20m15.940402399s) [noise=true , address=145.79.227.12:4767   , piece_id=OD3GGVFFAURCAPQ6VUAQBARGS6QEGJWS2QO3P4Q7FCSATLVI4KVA, seg_pos={0 0}, upload=525314]: Uploading piece...
Dec 30 14:49:17.770163792 (599h20m17.641246827s) [upload=525314]: Next returned nil segment
Dec 30 14:49:17.770178630 (599h20m17.641261665s) [upload=525314]: Waiting for error group managing segments...
Dec 30 14:49:18.050941885 (599h20m17.92202492s) [noise=true , address=145.79.227.9:4367    , piece_id=AHAOE6LGGXK76FXMHIEYQ5DFOI6Z2IB5PNSPVA53H25RPBJMDERQ, seg_pos={0 0}, upload=525314]: Done uploading piece. err: <nil>
Dec 30 14:49:18.053881252 (599h20m17.924964297s) [noise=true , address=145.79.227.2:4115    , piece_id=CVRG3KYICA6HRB72G5ZFLYXAVU2VBE7LSHWDUPXJ5ZI2CEQ646JQ, seg_pos={0 0}, upload=525314]: Done uploading piece. err: <nil>
Dec 30 14:49:18.057333535 (599h20m17.92841658s) [noise=true , address=145.79.227.50:4375   , piece_id=2QOU365LVL3EOSNAB55FFBLUMH7W7F6BXV4ZY4ICSMMFO626CZGQ, seg_pos={0 0}, upload=525314]: Done uploading piece. err: <nil>
Dec 30 14:49:18.057516063 (599h20m17.928599098s) [noise=true , address=145.79.227.12:4767   , piece_id=OD3GGVFFAURCAPQ6VUAQBARGS6QEGJWS2QO3P4Q7FCSATLVI4KVA, seg_pos={0 0}, upload=525314]: Done uploading piece. err: <nil>
Dec 30 14:49:18.073308103 (599h20m17.944391138s) [noise=true , address=145.79.227.11:4732   , piece_id=BQCXKQ2LKP5SSWQPDZZFHG4YJWKVL7ZKYNCPVSV324M2WGPDSHIA, seg_pos={0 0}, upload=525314]: Done uploading piece. err: <nil>
Dec 30 14:49:18.073609297 (599h20m17.944692342s) [noise=true , address=145.79.227.24:4581   , piece_id=7AJERH43PANL5BEBYVP2ZJ4LWGYYI7XD2UGPFMMQOSWHNOCSJZ5Q, seg_pos={0 0}, upload=525314]: Done uploading piece. err: <nil>
Dec 30 14:49:18.080166343 (599h20m17.951249378s) [noise=true , address=145.79.227.5:4223    , piece_id=UABOB6ROSJJJJSUVZ3BSFLSHBTIAMDK2W2JN5EIXDTO2FEKWCC7Q, seg_pos={0 0}, upload=525314]: Done uploading piece. err: <nil>
Dec 30 14:49:18.083336168 (599h20m17.954419203s) [noise=true , address=145.79.227.32:4836   , piece_id=G5L7AXETN2N33UUWO7YBWFRO4HQLB6AI77C3FXAFRD45LOFHWYBA, seg_pos={0 0}, upload=525314]: Done uploading piece. err: <nil>
Dec 30 14:49:18.134973794 (599h20m18.006056829s) [noise=true , address=145.79.227.36:4519   , piece_id=YNDF6NPMJQ5VUYXVLWRIAD3SJLYOKKAGLC45WURNSBHZ3SLRE2AQ, seg_pos={0 0}, upload=525314]: Done uploading piece. err: <nil>
Dec 30 14:49:18.142716637 (599h20m18.013799682s) [noise=true , address=145.79.227.8:4330    , piece_id=TFARMBVGBDUVJ33PNJW5OO27NW7T6COI55IA4U3I6QX6YW3QUOQA, seg_pos={0 0}, upload=525314]: Done uploading piece. err: <nil>
Dec 30 14:49:18.306226796 (599h20m18.177309831s) [noise=true , address=77.90.31.10:4517     , piece_id=AN3KCDQVIR4XVSZ6DF6IAPBVKLXTWPLNNUMELRUNX7HHFPELYI7Q, seg_pos={0 0}, upload=525314]: Done uploading piece. err: <nil>
Dec 30 14:49:18.353636440 (599h20m18.224719505s) [noise=true , address=77.90.31.19:4844     , piece_id=EVZOFDUMQLTTPS4X5IQEZTXE5L6GUNYXESGBGXLY3MJEACE4CAWA, seg_pos={0 0}, upload=525314]: Done uploading piece. err: <nil>
Dec 30 14:49:18.361358624 (599h20m18.23244166s) [noise=true , address=77.90.31.12:4590     , piece_id=BHUJU7RKWLWS63DGJDWTUFDG4DHNZGSLGJ7JXW6Q6TQ4BP33MXDQ, seg_pos={0 0}, upload=525314]: Done uploading piece. err: <nil>
Dec 30 14:49:18.361380135 (599h20m18.23246317s) [seg_pos={0 0}, upload=525314]: Segment reached optimal threshold of 13 pieces.
Dec 30 14:49:18.361641393 (599h20m18.232724428s) [noise=true , address=45.91.116.6:4976     , piece_id=PJUSIJPXPT353JLGPDBPV3WWGLMI6S2UMXPEM43UUIERPYDXGWFA, seg_pos={0 0}, upload=525314]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: read tcp 188.166.225.170:46946->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 30 14:49:18.361648025 (599h20m18.23273107s) [noise=true , address=45.91.116.6:4976     , piece_id=PJUSIJPXPT353JLGPDBPV3WWGLMI6S2UMXPEM43UUIERPYDXGWFA, seg_pos={0 0}, upload=525314]: Long tail cancellation due to optimal threshold
Dec 30 14:49:18.361704142 (599h20m18.232787178s) [noise=true , address=45.91.116.7:4012     , piece_id=V5GP7DRHXY63ADQLGTKEIDC7OGGR6FVLOGLAZV4FEF65PKVYP5UQ, seg_pos={0 0}, upload=525314]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF
Dec 30 14:49:18.361718730 (599h20m18.232801765s) [noise=true , address=45.91.116.7:4012     , piece_id=V5GP7DRHXY63ADQLGTKEIDC7OGGR6FVLOGLAZV4FEF65PKVYP5UQ, seg_pos={0 0}, upload=525314]: Long tail cancellation due to optimal threshold
Dec 30 14:49:18.361918621 (599h20m18.233001656s) [noise=true , address=77.90.31.21:4878     , piece_id=PGA6V26MV4NYZA6EWNUF2GE65UBK2ZAOG23T7B2UCEH7G2KC2E4Q, seg_pos={0 0}, upload=525314]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF
Dec 30 14:49:18.361926365 (599h20m18.2330094s) [noise=true , address=77.90.31.21:4878     , piece_id=PGA6V26MV4NYZA6EWNUF2GE65UBK2ZAOG23T7B2UCEH7G2KC2E4Q, seg_pos={0 0}, upload=525314]: Long tail cancellation due to optimal threshold
Dec 30 14:49:18.361945422 (599h20m18.233028457s) [seg_pos={0 0}, upload=525314]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 14:49:18.362048478 (599h20m18.233131513s) [upload=525314]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 14:49:18.362063186 (599h20m18.233146231s) [upload=525314]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 14:49:18.657997999 (599h20m18.529081034s) [upload=525314]: Done uploading segments...
Dec 30 14:49:18.658033917 (599h20m18.529116952s) [upload=525314]: Upload finished. err: <nil>
Duration: 5.94351075s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)
antiblip: 
3 out of 60 is failed (threshold=3) [.X.X........................................X...............]
duration: 
25.684318339s
trace: 
https://jaeger.dev.storj.io/trace/71938bc9cbb32ff3
FAIL (2025-12-30 14:49:50 UTC) 
orbiter warning: context deadline exceeded
Duration: 25.684332045s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)
antiblip: 
1 out of 60 is failed (threshold=3) [............................................X...............]
duration: 
8.173018353s
trace: 
https://jaeger.dev.storj.io/trace/414189176d03adca
FAIL (2025-12-30 14:49:46 UTC) 
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 8.17307961s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
17 out of 60 is failed (threshold=3) [.X.....X...X....X......X.......X..X....X..X...X..XXX..XX..XX]
duration: 
30.003062907s
trace: 
https://jaeger.dev.storj.io/trace/688751829c2f4ed5
FAIL (2025-12-30 14:49:45 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: AN3KCDQVIR4XVSZ6DF6IAPBVKLXTWPLNNUMELRUNX7HHFPELYI7Q) context deadline exceeded; piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: BHUJU7RKWLWS63DGJDWTUFDG4DHNZGSLGJ7JXW6Q6TQ4BP33MXDQ) context deadline exceeded
======LOGS======
Dec 30 14:49:15.821133341 (599h20m15.692216386s) [upload=525314]: Starting upload
Dec 30 14:49:15.821213894 (599h20m15.692296939s) [upload=525314]: Done starting upload
Dec 30 14:49:15.821291121 (599h20m15.692374166s) [upload=525314]: Uploading segments...
Dec 30 14:49:15.821470293 (599h20m15.692553328s) [upload=525314]: Got next segment. Inline: false
Dec 30 14:49:15.821510500 (599h20m15.692593535s) [upload=525314]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 14:49:15.821515269 (599h20m15.692598304s) [upload=525314]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 14:49:16.064550864 (599h20m15.935633909s) [seg_pos={0 0}, upload=525314]: Begin upload segment...
Dec 30 14:49:16.064606450 (599h20m15.935689485s) [seg_pos={0 0}, upload=525314]: Done begin upload segment.
Dec 30 14:49:16.064632299 (599h20m15.935715335s) [noise=true , address=145.79.227.11:4732   , piece_id=BQCXKQ2LKP5SSWQPDZZFHG4YJWKVL7ZKYNCPVSV324M2WGPDSHIA, seg_pos={0 0}, upload=525314]: Uploading piece...
Dec 30 14:49:16.064681142 (599h20m15.935764177s) [noise=true , address=145.79.227.2:4115    , piece_id=CVRG3KYICA6HRB72G5ZFLYXAVU2VBE7LSHWDUPXJ5ZI2CEQ646JQ, seg_pos={0 0}, upload=525314]: Uploading piece...
Dec 30 14:49:16.064775853 (599h20m15.935858898s) [noise=true , address=145.79.227.32:4836   , piece_id=G5L7AXETN2N33UUWO7YBWFRO4HQLB6AI77C3FXAFRD45LOFHWYBA, seg_pos={0 0}, upload=525314]: Uploading piece...
Dec 30 14:49:16.065109658 (599h20m15.936192693s) [noise=true , address=145.79.227.9:4367    , piece_id=AHAOE6LGGXK76FXMHIEYQ5DFOI6Z2IB5PNSPVA53H25RPBJMDERQ, seg_pos={0 0}, upload=525314]: Uploading piece...
Dec 30 14:49:16.065931713 (599h20m15.937014749s) [noise=true , address=145.79.227.24:4581   , piece_id=7AJERH43PANL5BEBYVP2ZJ4LWGYYI7XD2UGPFMMQOSWHNOCSJZ5Q, seg_pos={0 0}, upload=525314]: Uploading piece...
Dec 30 14:49:16.066105454 (599h20m15.937188489s) [noise=true , address=145.79.227.8:4330    , piece_id=TFARMBVGBDUVJ33PNJW5OO27NW7T6COI55IA4U3I6QX6YW3QUOQA, seg_pos={0 0}, upload=525314]: Uploading piece...
Dec 30 14:49:16.066246322 (599h20m15.937329368s) [noise=true , address=77.90.31.19:4844     , piece_id=EVZOFDUMQLTTPS4X5IQEZTXE5L6GUNYXESGBGXLY3MJEACE4CAWA, seg_pos={0 0}, upload=525314]: Uploading piece...
Dec 30 14:49:16.066560911 (599h20m15.937643957s) [noise=true , address=45.91.116.6:4976     , piece_id=PJUSIJPXPT353JLGPDBPV3WWGLMI6S2UMXPEM43UUIERPYDXGWFA, seg_pos={0 0}, upload=525314]: Uploading piece...
Dec 30 14:49:16.067090730 (599h20m15.938173765s) [noise=true , address=77.90.31.21:4878     , piece_id=PGA6V26MV4NYZA6EWNUF2GE65UBK2ZAOG23T7B2UCEH7G2KC2E4Q, seg_pos={0 0}, upload=525314]: Uploading piece...
Dec 30 14:49:16.067698037 (599h20m15.938781082s) [noise=true , address=145.79.227.36:4519   , piece_id=YNDF6NPMJQ5VUYXVLWRIAD3SJLYOKKAGLC45WURNSBHZ3SLRE2AQ, seg_pos={0 0}, upload=525314]: Uploading piece...
Dec 30 14:49:16.068034968 (599h20m15.939118003s) [noise=true , address=77.90.31.10:4517     , piece_id=AN3KCDQVIR4XVSZ6DF6IAPBVKLXTWPLNNUMELRUNX7HHFPELYI7Q, seg_pos={0 0}, upload=525314]: Uploading piece...
Dec 30 14:49:16.068256460 (599h20m15.939339505s) [noise=true , address=145.79.227.5:4223    , piece_id=UABOB6ROSJJJJSUVZ3BSFLSHBTIAMDK2W2JN5EIXDTO2FEKWCC7Q, seg_pos={0 0}, upload=525314]: Uploading piece...
Dec 30 14:49:16.068670930 (599h20m15.939753965s) [noise=true , address=45.91.116.7:4012     , piece_id=V5GP7DRHXY63ADQLGTKEIDC7OGGR6FVLOGLAZV4FEF65PKVYP5UQ, seg_pos={0 0}, upload=525314]: Uploading piece...
Dec 30 14:49:16.068792280 (599h20m15.939875325s) [noise=true , address=77.90.31.12:4590     , piece_id=BHUJU7RKWLWS63DGJDWTUFDG4DHNZGSLGJ7JXW6Q6TQ4BP33MXDQ, seg_pos={0 0}, upload=525314]: Uploading piece...
Dec 30 14:49:16.069004735 (599h20m15.94008778s) [noise=true , address=145.79.227.50:4375   , piece_id=2QOU365LVL3EOSNAB55FFBLUMH7W7F6BXV4ZY4ICSMMFO626CZGQ, seg_pos={0 0}, upload=525314]: Uploading piece...
Dec 30 14:49:16.069319354 (599h20m15.940402399s) [noise=true , address=145.79.227.12:4767   , piece_id=OD3GGVFFAURCAPQ6VUAQBARGS6QEGJWS2QO3P4Q7FCSATLVI4KVA, seg_pos={0 0}, upload=525314]: Uploading piece...
Dec 30 14:49:17.770163792 (599h20m17.641246827s) [upload=525314]: Next returned nil segment
Dec 30 14:49:17.770178630 (599h20m17.641261665s) [upload=525314]: Waiting for error group managing segments...
Dec 30 14:49:18.050941885 (599h20m17.92202492s) [noise=true , address=145.79.227.9:4367    , piece_id=AHAOE6LGGXK76FXMHIEYQ5DFOI6Z2IB5PNSPVA53H25RPBJMDERQ, seg_pos={0 0}, upload=525314]: Done uploading piece. err: <nil>
Dec 30 14:49:18.053881252 (599h20m17.924964297s) [noise=true , address=145.79.227.2:4115    , piece_id=CVRG3KYICA6HRB72G5ZFLYXAVU2VBE7LSHWDUPXJ5ZI2CEQ646JQ, seg_pos={0 0}, upload=525314]: Done uploading piece. err: <nil>
Dec 30 14:49:18.057333535 (599h20m17.92841658s) [noise=true , address=145.79.227.50:4375   , piece_id=2QOU365LVL3EOSNAB55FFBLUMH7W7F6BXV4ZY4ICSMMFO626CZGQ, seg_pos={0 0}, upload=525314]: Done uploading piece. err: <nil>
Dec 30 14:49:18.057516063 (599h20m17.928599098s) [noise=true , address=145.79.227.12:4767   , piece_id=OD3GGVFFAURCAPQ6VUAQBARGS6QEGJWS2QO3P4Q7FCSATLVI4KVA, seg_pos={0 0}, upload=525314]: Done uploading piece. err: <nil>
Dec 30 14:49:18.073308103 (599h20m17.944391138s) [noise=true , address=145.79.227.11:4732   , piece_id=BQCXKQ2LKP5SSWQPDZZFHG4YJWKVL7ZKYNCPVSV324M2WGPDSHIA, seg_pos={0 0}, upload=525314]: Done uploading piece. err: <nil>
Dec 30 14:49:18.073609297 (599h20m17.944692342s) [noise=true , address=145.79.227.24:4581   , piece_id=7AJERH43PANL5BEBYVP2ZJ4LWGYYI7XD2UGPFMMQOSWHNOCSJZ5Q, seg_pos={0 0}, upload=525314]: Done uploading piece. err: <nil>
Dec 30 14:49:18.080166343 (599h20m17.951249378s) [noise=true , address=145.79.227.5:4223    , piece_id=UABOB6ROSJJJJSUVZ3BSFLSHBTIAMDK2W2JN5EIXDTO2FEKWCC7Q, seg_pos={0 0}, upload=525314]: Done uploading piece. err: <nil>
Dec 30 14:49:18.083336168 (599h20m17.954419203s) [noise=true , address=145.79.227.32:4836   , piece_id=G5L7AXETN2N33UUWO7YBWFRO4HQLB6AI77C3FXAFRD45LOFHWYBA, seg_pos={0 0}, upload=525314]: Done uploading piece. err: <nil>
Dec 30 14:49:18.134973794 (599h20m18.006056829s) [noise=true , address=145.79.227.36:4519   , piece_id=YNDF6NPMJQ5VUYXVLWRIAD3SJLYOKKAGLC45WURNSBHZ3SLRE2AQ, seg_pos={0 0}, upload=525314]: Done uploading piece. err: <nil>
Dec 30 14:49:18.142716637 (599h20m18.013799682s) [noise=true , address=145.79.227.8:4330    , piece_id=TFARMBVGBDUVJ33PNJW5OO27NW7T6COI55IA4U3I6QX6YW3QUOQA, seg_pos={0 0}, upload=525314]: Done uploading piece. err: <nil>
Dec 30 14:49:18.306226796 (599h20m18.177309831s) [noise=true , address=77.90.31.10:4517     , piece_id=AN3KCDQVIR4XVSZ6DF6IAPBVKLXTWPLNNUMELRUNX7HHFPELYI7Q, seg_pos={0 0}, upload=525314]: Done uploading piece. err: <nil>
Dec 30 14:49:18.353636440 (599h20m18.224719505s) [noise=true , address=77.90.31.19:4844     , piece_id=EVZOFDUMQLTTPS4X5IQEZTXE5L6GUNYXESGBGXLY3MJEACE4CAWA, seg_pos={0 0}, upload=525314]: Done uploading piece. err: <nil>
Dec 30 14:49:18.361358624 (599h20m18.23244166s) [noise=true , address=77.90.31.12:4590     , piece_id=BHUJU7RKWLWS63DGJDWTUFDG4DHNZGSLGJ7JXW6Q6TQ4BP33MXDQ, seg_pos={0 0}, upload=525314]: Done uploading piece. err: <nil>
Dec 30 14:49:18.361380135 (599h20m18.23246317s) [seg_pos={0 0}, upload=525314]: Segment reached optimal threshold of 13 pieces.
Dec 30 14:49:18.361641393 (599h20m18.232724428s) [noise=true , address=45.91.116.6:4976     , piece_id=PJUSIJPXPT353JLGPDBPV3WWGLMI6S2UMXPEM43UUIERPYDXGWFA, seg_pos={0 0}, upload=525314]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: read tcp 188.166.225.170:46946->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 30 14:49:18.361648025 (599h20m18.23273107s) [noise=true , address=45.91.116.6:4976     , piece_id=PJUSIJPXPT353JLGPDBPV3WWGLMI6S2UMXPEM43UUIERPYDXGWFA, seg_pos={0 0}, upload=525314]: Long tail cancellation due to optimal threshold
Dec 30 14:49:18.361704142 (599h20m18.232787178s) [noise=true , address=45.91.116.7:4012     , piece_id=V5GP7DRHXY63ADQLGTKEIDC7OGGR6FVLOGLAZV4FEF65PKVYP5UQ, seg_pos={0 0}, upload=525314]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF
Dec 30 14:49:18.361718730 (599h20m18.232801765s) [noise=true , address=45.91.116.7:4012     , piece_id=V5GP7DRHXY63ADQLGTKEIDC7OGGR6FVLOGLAZV4FEF65PKVYP5UQ, seg_pos={0 0}, upload=525314]: Long tail cancellation due to optimal threshold
Dec 30 14:49:18.361918621 (599h20m18.233001656s) [noise=true , address=77.90.31.21:4878     , piece_id=PGA6V26MV4NYZA6EWNUF2GE65UBK2ZAOG23T7B2UCEH7G2KC2E4Q, seg_pos={0 0}, upload=525314]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF
Dec 30 14:49:18.361926365 (599h20m18.2330094s) [noise=true , address=77.90.31.21:4878     , piece_id=PGA6V26MV4NYZA6EWNUF2GE65UBK2ZAOG23T7B2UCEH7G2KC2E4Q, seg_pos={0 0}, upload=525314]: Long tail cancellation due to optimal threshold
Dec 30 14:49:18.361945422 (599h20m18.233028457s) [seg_pos={0 0}, upload=525314]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 14:49:18.362048478 (599h20m18.233131513s) [upload=525314]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 14:49:18.362063186 (599h20m18.233146231s) [upload=525314]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 14:49:18.657997999 (599h20m18.529081034s) [upload=525314]: Done uploading segments...
Dec 30 14:49:18.658033917 (599h20m18.529116952s) [upload=525314]: Upload finished. err: <nil>
Duration: 30.003082945s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
14 out of 60 is failed (threshold=3) [.XXX.....X......X......X........X.X.X..X..X.....XXX.........]
duration: 
17.654545107s
trace: 
https://jaeger.dev.storj.io/trace/41ecdbaa0176ef93
FAIL (2025-12-30 14:49:45 UTC) 
orbiter warning: context deadline exceeded
Duration: 17.654570886s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [..............................X...........................X.]
duration: 
34.342172528s
trace: 
https://jaeger.dev.storj.io/trace/328a692d1ab4d967
FAIL (2025-12-30 14:49:44 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 17ddUMj4ZejcjjWGzkGEM63vbCoEPnu7ePznzAPyzQofW8Q4gd, Piece ID: AP324FY2ZWKR4JNW7WET4XMLHNIR53YJHS4EYA7T2NFQMDAPJGOQ) context deadline exceeded; piecestore: (Node ID: 128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, Piece ID: SRPWR2M2GYYA3JOMGHMDZHHXI2FLDJCIQROQZAVFDTV6ASPDDVMA) context deadline exceeded; piecestore: (Node ID: 1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, Piece ID: 7RS2ACDDX55LLN75FUYF7YFC72ACCLFSKPGHBDAMC4K5RPZBEK2A) context deadline exceeded; piecestore: (Node ID: 12qXhyMbfXELW1Vg5z9NpqFX1fm3v2pGEaA56qSc8kHefPNNNYs, Piece ID: ZUQVNQBMWUN7JD5GNPBNIVDHHHDAJJH7DGWRJZ2GBYGQMF25ZBTQ) context deadline exceeded
======LOGS======
oise=true , address=145.79.227.43:4180   , piece_id=IQEZVMY7IRLS5VQSPUXNUZSP3XZZ67X4TDLDONUT4MVRNVNEAANQ, seg_pos={0 0}, upload=525299]: Uploading piece...
Dec 30 14:48:11.029459924 (599h19m10.900542959s) [noise=true , address=64.49.13.155:4920    , piece_id=D3YEBC7PPVGA5DCFWUWR6BFHV3N43WE4RPAS7LT7GX5XIRMLPEUQ, seg_pos={0 0}, upload=525299]: Uploading piece...
Dec 30 14:48:11.029880215 (599h19m10.90096326s) [noise=true , address=145.79.227.34:4935   , piece_id=C3EF7WUSWDMZKVMULEXZHMQOKRH2ODJVVHRKIV57GYH4INI5QIJA, seg_pos={0 0}, upload=525299]: Uploading piece...
Dec 30 14:48:11.030199834 (599h19m10.901282869s) [noise=true , address=145.79.227.49:4347   , piece_id=TILSKTAE3CDAL4UIYZK5POLHEAJCCI5PFJJX5SNEBYIJHP2TPFHQ, seg_pos={0 0}, upload=525299]: Uploading piece...
Dec 30 14:48:11.030300455 (599h19m10.90138349s) [noise=true , address=145.79.227.4:4210    , piece_id=KWBC7SM7JYHXLRPBOOXGU7QQKDJWRB2U3AOBDQCKLLTZZLLFXDMQ, seg_pos={0 0}, upload=525299]: Uploading piece...
Dec 30 14:48:11.030568916 (599h19m10.901651951s) [noise=true , address=77.90.31.14:4689     , piece_id=7RS2ACDDX55LLN75FUYF7YFC72ACCLFSKPGHBDAMC4K5RPZBEK2A, seg_pos={0 0}, upload=525299]: Uploading piece...
Dec 30 14:48:11.030753688 (599h19m10.901836724s) [noise=true , address=77.90.31.19:4860     , piece_id=AP324FY2ZWKR4JNW7WET4XMLHNIR53YJHS4EYA7T2NFQMDAPJGOQ, seg_pos={0 0}, upload=525299]: Uploading piece...
Dec 30 14:48:11.030910497 (599h19m10.901993532s) [noise=true , address=77.90.31.11:4586     , piece_id=XJDAQMNAIBFY6EICSLSIU52WIAIJKNPODKNYRWECS6CV7GELAMGQ, seg_pos={0 0}, upload=525299]: Uploading piece...
Dec 30 14:48:11.031175852 (599h19m10.902258887s) [noise=true , address=145.79.227.44:4212   , piece_id=QTY5RT75QRIUL7GBRB3V4LJK4JL4U4IN4RGZMBV7TOKZ75DOSQEQ, seg_pos={0 0}, upload=525299]: Uploading piece...
Dec 30 14:48:11.031220547 (599h19m10.902303592s) [noise=true , address=45.91.116.12:4252    , piece_id=ISEASO5OV7UBUPGR3B4CJJU3QCDYLC365PT4B4BKAIN3W7GTX4BQ, seg_pos={0 0}, upload=525299]: Uploading piece...
Dec 30 14:48:11.031518976 (599h19m10.902602011s) [noise=true , address=64.49.13.169:4709    , piece_id=PVOTCKOHNI7LLPOEZNQ7RHQADMYDGVVAD5HGLRGYDKAZ7W22VUFA, seg_pos={0 0}, upload=525299]: Uploading piece...
Dec 30 14:48:11.031659844 (599h19m10.902742879s) [noise=true , address=77.90.31.13:4633     , piece_id=ZUQVNQBMWUN7JD5GNPBNIVDHHHDAJJH7DGWRJZ2GBYGQMF25ZBTQ, seg_pos={0 0}, upload=525299]: Uploading piece...
Dec 30 14:48:11.031896624 (599h19m10.902979659s) [noise=true , address=77.90.31.18:4812     , piece_id=SRPWR2M2GYYA3JOMGHMDZHHXI2FLDJCIQROQZAVFDTV6ASPDDVMA, seg_pos={0 0}, upload=525299]: Uploading piece...
Dec 30 14:48:11.032080996 (599h19m10.903164031s) [noise=true , address=145.79.227.30:4769   , piece_id=WL2KVAFIDCIBEX6PGFVT5M4JAC525YC3MAU4SVVUMFS3FAB6QBNQ, seg_pos={0 0}, upload=525299]: Uploading piece...
Dec 30 14:48:11.032234387 (599h19m10.903317422s) [noise=true , address=145.79.227.35:4513   , piece_id=SABYA6SJYZ65IFUYFGSYU6UUS6QSXFCIQCT7BRSJM7BCWHQILYDA, seg_pos={0 0}, upload=525299]: Uploading piece...
Dec 30 14:48:11.032571650 (599h19m10.903654685s) [noise=true , address=64.49.13.137:4494    , piece_id=RRKIPPWSAWV7QILA4PGQJ2IKAGJ6BSMP7OPFRQJJK4UWWWQNXVZA, seg_pos={0 0}, upload=525299]: Uploading piece...
Dec 30 14:48:12.423070865 (599h19m12.2941539s) [upload=525299]: Next returned nil segment
Dec 30 14:48:12.423093118 (599h19m12.294176153s) [upload=525299]: Waiting for error group managing segments...
Dec 30 14:48:12.831938369 (599h19m12.703021404s) [noise=true , address=64.49.13.27:4965     , piece_id=XUTKVFCA5BJNZIBJCBWZ75K3B3TCWSZ5U5ESHATA6K376Z2YNXKA, seg_pos={0 0}, upload=525299]: Done uploading piece. err: <nil>
Dec 30 14:48:12.868364350 (599h19m12.739447395s) [noise=true , address=64.49.13.137:4494    , piece_id=RRKIPPWSAWV7QILA4PGQJ2IKAGJ6BSMP7OPFRQJJK4UWWWQNXVZA, seg_pos={0 0}, upload=525299]: Done uploading piece. err: <nil>
Dec 30 14:48:12.959516487 (599h19m12.830599532s) [noise=true , address=64.49.13.155:4920    , piece_id=D3YEBC7PPVGA5DCFWUWR6BFHV3N43WE4RPAS7LT7GX5XIRMLPEUQ, seg_pos={0 0}, upload=525299]: Done uploading piece. err: <nil>
Dec 30 14:48:12.975446991 (599h19m12.846530026s) [noise=true , address=145.79.227.4:4210    , piece_id=KWBC7SM7JYHXLRPBOOXGU7QQKDJWRB2U3AOBDQCKLLTZZLLFXDMQ, seg_pos={0 0}, upload=525299]: Done uploading piece. err: <nil>
Dec 30 14:48:13.079109379 (599h19m12.950192414s) [noise=true , address=77.90.31.19:4860     , piece_id=AP324FY2ZWKR4JNW7WET4XMLHNIR53YJHS4EYA7T2NFQMDAPJGOQ, seg_pos={0 0}, upload=525299]: Done uploading piece. err: <nil>
Dec 30 14:48:13.091745701 (599h19m12.962828746s) [noise=true , address=77.90.31.13:4633     , piece_id=ZUQVNQBMWUN7JD5GNPBNIVDHHHDAJJH7DGWRJZ2GBYGQMF25ZBTQ, seg_pos={0 0}, upload=525299]: Done uploading piece. err: <nil>
Dec 30 14:48:13.101549980 (599h19m12.972633025s) [noise=true , address=77.90.31.18:4812     , piece_id=SRPWR2M2GYYA3JOMGHMDZHHXI2FLDJCIQROQZAVFDTV6ASPDDVMA, seg_pos={0 0}, upload=525299]: Done uploading piece. err: <nil>
Dec 30 14:48:13.270615245 (599h19m13.14169828s) [noise=true , address=45.91.116.12:4252    , piece_id=ISEASO5OV7UBUPGR3B4CJJU3QCDYLC365PT4B4BKAIN3W7GTX4BQ, seg_pos={0 0}, upload=525299]: Done uploading piece. err: <nil>
Dec 30 14:48:13.396635416 (599h19m13.267718461s) [noise=true , address=77.90.31.11:4586     , piece_id=XJDAQMNAIBFY6EICSLSIU52WIAIJKNPODKNYRWECS6CV7GELAMGQ, seg_pos={0 0}, upload=525299]: Done uploading piece. err: <nil>
Dec 30 14:48:13.735330842 (599h19m13.606413877s) [noise=true , address=145.79.227.49:4347   , piece_id=TILSKTAE3CDAL4UIYZK5POLHEAJCCI5PFJJX5SNEBYIJHP2TPFHQ, seg_pos={0 0}, upload=525299]: Done uploading piece. err: <nil>
Dec 30 14:48:13.775816923 (599h19m13.646899958s) [noise=true , address=145.79.227.30:4769   , piece_id=WL2KVAFIDCIBEX6PGFVT5M4JAC525YC3MAU4SVVUMFS3FAB6QBNQ, seg_pos={0 0}, upload=525299]: Done uploading piece. err: <nil>
Dec 30 14:48:13.782446286 (599h19m13.653529321s) [noise=true , address=145.79.227.43:4180   , piece_id=IQEZVMY7IRLS5VQSPUXNUZSP3XZZ67X4TDLDONUT4MVRNVNEAANQ, seg_pos={0 0}, upload=525299]: Done uploading piece. err: <nil>
Dec 30 14:48:13.833466567 (599h19m13.704549602s) [noise=true , address=145.79.227.44:4212   , piece_id=QTY5RT75QRIUL7GBRB3V4LJK4JL4U4IN4RGZMBV7TOKZ75DOSQEQ, seg_pos={0 0}, upload=525299]: Done uploading piece. err: <nil>
Dec 30 14:48:13.934059641 (599h19m13.805142676s) [noise=true , address=77.90.31.14:4689     , piece_id=7RS2ACDDX55LLN75FUYF7YFC72ACCLFSKPGHBDAMC4K5RPZBEK2A, seg_pos={0 0}, upload=525299]: Done uploading piece. err: <nil>
Dec 30 14:48:13.934166574 (599h19m13.805249609s) [seg_pos={0 0}, upload=525299]: Segment reached optimal threshold of 14 pieces.
Dec 30 14:48:13.934461506 (599h19m13.805544551s) [noise=true , address=145.79.227.35:4513   , piece_id=SABYA6SJYZ65IFUYFGSYU6UUS6QSXFCIQCT7BRSJM7BCWHQILYDA, seg_pos={0 0}, upload=525299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pyyW8Gdqg6j9YGMtCUBhFnoNXryvPMUy19x815dp2HV3Xqr8h): protocol: expected piece hash; context canceled; EOF
Dec 30 14:48:13.934472587 (599h19m13.805555622s) [noise=true , address=145.79.227.35:4513   , piece_id=SABYA6SJYZ65IFUYFGSYU6UUS6QSXFCIQCT7BRSJM7BCWHQILYDA, seg_pos={0 0}, upload=525299]: Long tail cancellation due to optimal threshold
Dec 30 14:48:13.934771517 (599h19m13.805854552s) [noise=true , address=77.90.31.12:4620     , piece_id=TXXA7FP2AVAOXTQ7B4Y2XCCCDW6HBJQ3BY6EMOV6X4IHWNUHQ7UQ, seg_pos={0 0}, upload=525299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe): protocol: expected piece hash; context canceled; EOF
Dec 30 14:48:13.934779772 (599h19m13.805862807s) [noise=true , address=77.90.31.12:4620     , piece_id=TXXA7FP2AVAOXTQ7B4Y2XCCCDW6HBJQ3BY6EMOV6X4IHWNUHQ7UQ, seg_pos={0 0}, upload=525299]: Long tail cancellation due to optimal threshold
Dec 30 14:48:13.935018687 (599h19m13.806101732s) [noise=true , address=145.79.227.34:4935   , piece_id=C3EF7WUSWDMZKVMULEXZHMQOKRH2ODJVVHRKIV57GYH4INI5QIJA, seg_pos={0 0}, upload=525299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ks2kQWUmV377HgGbyoTt96CTyK1P1nH8QtZQzh3xvnFCYaD8X): protocol: expected piece hash; context canceled; EOF
Dec 30 14:48:13.935027324 (599h19m13.806110369s) [noise=true , address=145.79.227.34:4935   , piece_id=C3EF7WUSWDMZKVMULEXZHMQOKRH2ODJVVHRKIV57GYH4INI5QIJA, seg_pos={0 0}, upload=525299]: Long tail cancellation due to optimal threshold
Dec 30 14:48:13.935192078 (599h19m13.806275113s) [noise=true , address=64.49.13.169:4709    , piece_id=PVOTCKOHNI7LLPOEZNQ7RHQADMYDGVVAD5HGLRGYDKAZ7W22VUFA, seg_pos={0 0}, upload=525299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:55418->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:55418->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55418->64.49.13.169:4709: use of closed network connection
Dec 30 14:48:13.935200634 (599h19m13.806283669s) [noise=true , address=64.49.13.169:4709    , piece_id=PVOTCKOHNI7LLPOEZNQ7RHQADMYDGVVAD5HGLRGYDKAZ7W22VUFA, seg_pos={0 0}, upload=525299]: Long tail cancellation due to optimal threshold
Dec 30 14:48:13.935254516 (599h19m13.806337561s) [seg_pos={0 0}, upload=525299]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 14:48:13.935335981 (599h19m13.806419027s) [upload=525299]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 14:48:13.935342193 (599h19m13.806425238s) [upload=525299]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 14:48:14.224449907 (599h19m14.095532972s) [upload=525299]: Done uploading segments...
Dec 30 14:48:14.224476296 (599h19m14.095559332s) [upload=525299]: Upload finished. err: <nil>
Duration: 34.342190883s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
47 out of 60 is failed (threshold=6) [XXXXXXX.XXXXXXXXXXXXXXX.X..XX.XXXX.XXXXXXX.X..XX.X...XXXXXXX]
duration: 
5.001595878s
trace: 
https://jaeger.dev.storj.io/trace/6762c0bbed191862
FAIL (2025-12-30 14:49:44 UTC) 
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.00162425s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)
antiblip: 
1 out of 60 is failed (threshold=3) [.............................................X..............]
duration: 
5.488525771s
trace: 
https://jaeger.dev.storj.io/trace/414189176d03adca
FAIL (2025-12-30 14:49:11 UTC) 
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 5.488548644s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [...............................X...........................X]
duration: 
1m0.00254813s
trace: 
https://jaeger.dev.storj.io/trace/328a692d1ab4d967
FAIL (2025-12-30 14:49:10 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 17ddUMj4ZejcjjWGzkGEM63vbCoEPnu7ePznzAPyzQofW8Q4gd, Piece ID: AP324FY2ZWKR4JNW7WET4XMLHNIR53YJHS4EYA7T2NFQMDAPJGOQ) context deadline exceeded; piecestore: (Node ID: 128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, Piece ID: SRPWR2M2GYYA3JOMGHMDZHHXI2FLDJCIQROQZAVFDTV6ASPDDVMA) context deadline exceeded; piecestore: (Node ID: 1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, Piece ID: 7RS2ACDDX55LLN75FUYF7YFC72ACCLFSKPGHBDAMC4K5RPZBEK2A) context deadline exceeded; piecestore: (Node ID: 12qXhyMbfXELW1Vg5z9NpqFX1fm3v2pGEaA56qSc8kHefPNNNYs, Piece ID: ZUQVNQBMWUN7JD5GNPBNIVDHHHDAJJH7DGWRJZ2GBYGQMF25ZBTQ) context deadline exceeded
======LOGS======
oise=true , address=145.79.227.43:4180   , piece_id=IQEZVMY7IRLS5VQSPUXNUZSP3XZZ67X4TDLDONUT4MVRNVNEAANQ, seg_pos={0 0}, upload=525299]: Uploading piece...
Dec 30 14:48:11.029459924 (599h19m10.900542959s) [noise=true , address=64.49.13.155:4920    , piece_id=D3YEBC7PPVGA5DCFWUWR6BFHV3N43WE4RPAS7LT7GX5XIRMLPEUQ, seg_pos={0 0}, upload=525299]: Uploading piece...
Dec 30 14:48:11.029880215 (599h19m10.90096326s) [noise=true , address=145.79.227.34:4935   , piece_id=C3EF7WUSWDMZKVMULEXZHMQOKRH2ODJVVHRKIV57GYH4INI5QIJA, seg_pos={0 0}, upload=525299]: Uploading piece...
Dec 30 14:48:11.030199834 (599h19m10.901282869s) [noise=true , address=145.79.227.49:4347   , piece_id=TILSKTAE3CDAL4UIYZK5POLHEAJCCI5PFJJX5SNEBYIJHP2TPFHQ, seg_pos={0 0}, upload=525299]: Uploading piece...
Dec 30 14:48:11.030300455 (599h19m10.90138349s) [noise=true , address=145.79.227.4:4210    , piece_id=KWBC7SM7JYHXLRPBOOXGU7QQKDJWRB2U3AOBDQCKLLTZZLLFXDMQ, seg_pos={0 0}, upload=525299]: Uploading piece...
Dec 30 14:48:11.030568916 (599h19m10.901651951s) [noise=true , address=77.90.31.14:4689     , piece_id=7RS2ACDDX55LLN75FUYF7YFC72ACCLFSKPGHBDAMC4K5RPZBEK2A, seg_pos={0 0}, upload=525299]: Uploading piece...
Dec 30 14:48:11.030753688 (599h19m10.901836724s) [noise=true , address=77.90.31.19:4860     , piece_id=AP324FY2ZWKR4JNW7WET4XMLHNIR53YJHS4EYA7T2NFQMDAPJGOQ, seg_pos={0 0}, upload=525299]: Uploading piece...
Dec 30 14:48:11.030910497 (599h19m10.901993532s) [noise=true , address=77.90.31.11:4586     , piece_id=XJDAQMNAIBFY6EICSLSIU52WIAIJKNPODKNYRWECS6CV7GELAMGQ, seg_pos={0 0}, upload=525299]: Uploading piece...
Dec 30 14:48:11.031175852 (599h19m10.902258887s) [noise=true , address=145.79.227.44:4212   , piece_id=QTY5RT75QRIUL7GBRB3V4LJK4JL4U4IN4RGZMBV7TOKZ75DOSQEQ, seg_pos={0 0}, upload=525299]: Uploading piece...
Dec 30 14:48:11.031220547 (599h19m10.902303592s) [noise=true , address=45.91.116.12:4252    , piece_id=ISEASO5OV7UBUPGR3B4CJJU3QCDYLC365PT4B4BKAIN3W7GTX4BQ, seg_pos={0 0}, upload=525299]: Uploading piece...
Dec 30 14:48:11.031518976 (599h19m10.902602011s) [noise=true , address=64.49.13.169:4709    , piece_id=PVOTCKOHNI7LLPOEZNQ7RHQADMYDGVVAD5HGLRGYDKAZ7W22VUFA, seg_pos={0 0}, upload=525299]: Uploading piece...
Dec 30 14:48:11.031659844 (599h19m10.902742879s) [noise=true , address=77.90.31.13:4633     , piece_id=ZUQVNQBMWUN7JD5GNPBNIVDHHHDAJJH7DGWRJZ2GBYGQMF25ZBTQ, seg_pos={0 0}, upload=525299]: Uploading piece...
Dec 30 14:48:11.031896624 (599h19m10.902979659s) [noise=true , address=77.90.31.18:4812     , piece_id=SRPWR2M2GYYA3JOMGHMDZHHXI2FLDJCIQROQZAVFDTV6ASPDDVMA, seg_pos={0 0}, upload=525299]: Uploading piece...
Dec 30 14:48:11.032080996 (599h19m10.903164031s) [noise=true , address=145.79.227.30:4769   , piece_id=WL2KVAFIDCIBEX6PGFVT5M4JAC525YC3MAU4SVVUMFS3FAB6QBNQ, seg_pos={0 0}, upload=525299]: Uploading piece...
Dec 30 14:48:11.032234387 (599h19m10.903317422s) [noise=true , address=145.79.227.35:4513   , piece_id=SABYA6SJYZ65IFUYFGSYU6UUS6QSXFCIQCT7BRSJM7BCWHQILYDA, seg_pos={0 0}, upload=525299]: Uploading piece...
Dec 30 14:48:11.032571650 (599h19m10.903654685s) [noise=true , address=64.49.13.137:4494    , piece_id=RRKIPPWSAWV7QILA4PGQJ2IKAGJ6BSMP7OPFRQJJK4UWWWQNXVZA, seg_pos={0 0}, upload=525299]: Uploading piece...
Dec 30 14:48:12.423070865 (599h19m12.2941539s) [upload=525299]: Next returned nil segment
Dec 30 14:48:12.423093118 (599h19m12.294176153s) [upload=525299]: Waiting for error group managing segments...
Dec 30 14:48:12.831938369 (599h19m12.703021404s) [noise=true , address=64.49.13.27:4965     , piece_id=XUTKVFCA5BJNZIBJCBWZ75K3B3TCWSZ5U5ESHATA6K376Z2YNXKA, seg_pos={0 0}, upload=525299]: Done uploading piece. err: <nil>
Dec 30 14:48:12.868364350 (599h19m12.739447395s) [noise=true , address=64.49.13.137:4494    , piece_id=RRKIPPWSAWV7QILA4PGQJ2IKAGJ6BSMP7OPFRQJJK4UWWWQNXVZA, seg_pos={0 0}, upload=525299]: Done uploading piece. err: <nil>
Dec 30 14:48:12.959516487 (599h19m12.830599532s) [noise=true , address=64.49.13.155:4920    , piece_id=D3YEBC7PPVGA5DCFWUWR6BFHV3N43WE4RPAS7LT7GX5XIRMLPEUQ, seg_pos={0 0}, upload=525299]: Done uploading piece. err: <nil>
Dec 30 14:48:12.975446991 (599h19m12.846530026s) [noise=true , address=145.79.227.4:4210    , piece_id=KWBC7SM7JYHXLRPBOOXGU7QQKDJWRB2U3AOBDQCKLLTZZLLFXDMQ, seg_pos={0 0}, upload=525299]: Done uploading piece. err: <nil>
Dec 30 14:48:13.079109379 (599h19m12.950192414s) [noise=true , address=77.90.31.19:4860     , piece_id=AP324FY2ZWKR4JNW7WET4XMLHNIR53YJHS4EYA7T2NFQMDAPJGOQ, seg_pos={0 0}, upload=525299]: Done uploading piece. err: <nil>
Dec 30 14:48:13.091745701 (599h19m12.962828746s) [noise=true , address=77.90.31.13:4633     , piece_id=ZUQVNQBMWUN7JD5GNPBNIVDHHHDAJJH7DGWRJZ2GBYGQMF25ZBTQ, seg_pos={0 0}, upload=525299]: Done uploading piece. err: <nil>
Dec 30 14:48:13.101549980 (599h19m12.972633025s) [noise=true , address=77.90.31.18:4812     , piece_id=SRPWR2M2GYYA3JOMGHMDZHHXI2FLDJCIQROQZAVFDTV6ASPDDVMA, seg_pos={0 0}, upload=525299]: Done uploading piece. err: <nil>
Dec 30 14:48:13.270615245 (599h19m13.14169828s) [noise=true , address=45.91.116.12:4252    , piece_id=ISEASO5OV7UBUPGR3B4CJJU3QCDYLC365PT4B4BKAIN3W7GTX4BQ, seg_pos={0 0}, upload=525299]: Done uploading piece. err: <nil>
Dec 30 14:48:13.396635416 (599h19m13.267718461s) [noise=true , address=77.90.31.11:4586     , piece_id=XJDAQMNAIBFY6EICSLSIU52WIAIJKNPODKNYRWECS6CV7GELAMGQ, seg_pos={0 0}, upload=525299]: Done uploading piece. err: <nil>
Dec 30 14:48:13.735330842 (599h19m13.606413877s) [noise=true , address=145.79.227.49:4347   , piece_id=TILSKTAE3CDAL4UIYZK5POLHEAJCCI5PFJJX5SNEBYIJHP2TPFHQ, seg_pos={0 0}, upload=525299]: Done uploading piece. err: <nil>
Dec 30 14:48:13.775816923 (599h19m13.646899958s) [noise=true , address=145.79.227.30:4769   , piece_id=WL2KVAFIDCIBEX6PGFVT5M4JAC525YC3MAU4SVVUMFS3FAB6QBNQ, seg_pos={0 0}, upload=525299]: Done uploading piece. err: <nil>
Dec 30 14:48:13.782446286 (599h19m13.653529321s) [noise=true , address=145.79.227.43:4180   , piece_id=IQEZVMY7IRLS5VQSPUXNUZSP3XZZ67X4TDLDONUT4MVRNVNEAANQ, seg_pos={0 0}, upload=525299]: Done uploading piece. err: <nil>
Dec 30 14:48:13.833466567 (599h19m13.704549602s) [noise=true , address=145.79.227.44:4212   , piece_id=QTY5RT75QRIUL7GBRB3V4LJK4JL4U4IN4RGZMBV7TOKZ75DOSQEQ, seg_pos={0 0}, upload=525299]: Done uploading piece. err: <nil>
Dec 30 14:48:13.934059641 (599h19m13.805142676s) [noise=true , address=77.90.31.14:4689     , piece_id=7RS2ACDDX55LLN75FUYF7YFC72ACCLFSKPGHBDAMC4K5RPZBEK2A, seg_pos={0 0}, upload=525299]: Done uploading piece. err: <nil>
Dec 30 14:48:13.934166574 (599h19m13.805249609s) [seg_pos={0 0}, upload=525299]: Segment reached optimal threshold of 14 pieces.
Dec 30 14:48:13.934461506 (599h19m13.805544551s) [noise=true , address=145.79.227.35:4513   , piece_id=SABYA6SJYZ65IFUYFGSYU6UUS6QSXFCIQCT7BRSJM7BCWHQILYDA, seg_pos={0 0}, upload=525299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pyyW8Gdqg6j9YGMtCUBhFnoNXryvPMUy19x815dp2HV3Xqr8h): protocol: expected piece hash; context canceled; EOF
Dec 30 14:48:13.934472587 (599h19m13.805555622s) [noise=true , address=145.79.227.35:4513   , piece_id=SABYA6SJYZ65IFUYFGSYU6UUS6QSXFCIQCT7BRSJM7BCWHQILYDA, seg_pos={0 0}, upload=525299]: Long tail cancellation due to optimal threshold
Dec 30 14:48:13.934771517 (599h19m13.805854552s) [noise=true , address=77.90.31.12:4620     , piece_id=TXXA7FP2AVAOXTQ7B4Y2XCCCDW6HBJQ3BY6EMOV6X4IHWNUHQ7UQ, seg_pos={0 0}, upload=525299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe): protocol: expected piece hash; context canceled; EOF
Dec 30 14:48:13.934779772 (599h19m13.805862807s) [noise=true , address=77.90.31.12:4620     , piece_id=TXXA7FP2AVAOXTQ7B4Y2XCCCDW6HBJQ3BY6EMOV6X4IHWNUHQ7UQ, seg_pos={0 0}, upload=525299]: Long tail cancellation due to optimal threshold
Dec 30 14:48:13.935018687 (599h19m13.806101732s) [noise=true , address=145.79.227.34:4935   , piece_id=C3EF7WUSWDMZKVMULEXZHMQOKRH2ODJVVHRKIV57GYH4INI5QIJA, seg_pos={0 0}, upload=525299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ks2kQWUmV377HgGbyoTt96CTyK1P1nH8QtZQzh3xvnFCYaD8X): protocol: expected piece hash; context canceled; EOF
Dec 30 14:48:13.935027324 (599h19m13.806110369s) [noise=true , address=145.79.227.34:4935   , piece_id=C3EF7WUSWDMZKVMULEXZHMQOKRH2ODJVVHRKIV57GYH4INI5QIJA, seg_pos={0 0}, upload=525299]: Long tail cancellation due to optimal threshold
Dec 30 14:48:13.935192078 (599h19m13.806275113s) [noise=true , address=64.49.13.169:4709    , piece_id=PVOTCKOHNI7LLPOEZNQ7RHQADMYDGVVAD5HGLRGYDKAZ7W22VUFA, seg_pos={0 0}, upload=525299]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:55418->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:55418->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55418->64.49.13.169:4709: use of closed network connection
Dec 30 14:48:13.935200634 (599h19m13.806283669s) [noise=true , address=64.49.13.169:4709    , piece_id=PVOTCKOHNI7LLPOEZNQ7RHQADMYDGVVAD5HGLRGYDKAZ7W22VUFA, seg_pos={0 0}, upload=525299]: Long tail cancellation due to optimal threshold
Dec 30 14:48:13.935254516 (599h19m13.806337561s) [seg_pos={0 0}, upload=525299]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 14:48:13.935335981 (599h19m13.806419027s) [upload=525299]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 14:48:13.935342193 (599h19m13.806425238s) [upload=525299]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 14:48:14.224449907 (599h19m14.095532972s) [upload=525299]: Done uploading segments...
Dec 30 14:48:14.224476296 (599h19m14.095559332s) [upload=525299]: Upload finished. err: <nil>
Duration: 1m0.002570122s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
16 out of 60 is failed (threshold=3) [..X.....X...X....X......X.......X..X....X..X...X..XXX..XX..X]
duration: 
30.003201004s
trace: 
https://jaeger.dev.storj.io/trace/1b2386a42d79dfd2
FAIL (2025-12-30 14:49:09 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: 4EBP2YLPVM3Z5QVEA46WQFGEJ35BRH65KMJSE4GNOAR4AKQZBADA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: QBPAKBITBTU66GRCHIU47SNJ46L5DUAPA74NLYEFG7HD6N42QWEQ) context deadline exceeded
======LOGS======
 30 14:48:40.305459717 (599h19m40.176542752s) [upload=525305]: Starting upload
Dec 30 14:48:40.305538388 (599h19m40.176621423s) [upload=525305]: Done starting upload
Dec 30 14:48:40.305709394 (599h19m40.176792439s) [upload=525305]: Uploading segments...
Dec 30 14:48:40.305747195 (599h19m40.176830241s) [upload=525305]: Got next segment. Inline: false
Dec 30 14:48:40.305774988 (599h19m40.176858033s) [upload=525305]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 14:48:40.305778946 (599h19m40.176861981s) [upload=525305]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 14:48:40.570553825 (599h19m40.44163687s) [seg_pos={0 0}, upload=525305]: Begin upload segment...
Dec 30 14:48:40.570666700 (599h19m40.441749735s) [seg_pos={0 0}, upload=525305]: Done begin upload segment.
Dec 30 14:48:40.570708951 (599h19m40.441791986s) [noise=true , address=45.91.116.5:4299     , piece_id=QBPAKBITBTU66GRCHIU47SNJ46L5DUAPA74NLYEFG7HD6N42QWEQ, seg_pos={0 0}, upload=525305]: Uploading piece...
Dec 30 14:48:40.570760659 (599h19m40.441843724s) [noise=true , address=77.90.31.15:4700     , piece_id=5CB3GBBASEB34KIVBKULFLJGMTF4M5B2OXDVPZ4VM5NU5UU4TNDQ, seg_pos={0 0}, upload=525305]: Uploading piece...
Dec 30 14:48:40.572045055 (599h19m40.44312809s) [noise=true , address=145.79.227.17:4970   , piece_id=OJ5JPYFNHMDVGKSCSC4EHMINHI2BJDXYLOSUW2PYH2A647YVG7OQ, seg_pos={0 0}, upload=525305]: Uploading piece...
Dec 30 14:48:40.575253674 (599h19m40.446336709s) [noise=true , address=145.79.227.41:4087   , piece_id=GC7NQBAEZUVF7MUEBE6FRGTVM7YAHXGZTJV72WLVCL3ZOXES52CA, seg_pos={0 0}, upload=525305]: Uploading piece...
Dec 30 14:48:40.575264635 (599h19m40.44634767s) [noise=true , address=145.79.227.4:4185    , piece_id=AM2ET22HFDB5NS53OA46W2CDSQY5T7SIAGAK36BYPY2UGWDCIUQQ, seg_pos={0 0}, upload=525305]: Uploading piece...
Dec 30 14:48:40.576597123 (599h19m40.447680158s) [noise=true , address=145.79.227.3:4149    , piece_id=BGSRACZLQKJOKG3IDSHVKM5SGRQ4SNTAKWZOK2LLMVNBOJ5KVG5A, seg_pos={0 0}, upload=525305]: Uploading piece...
Dec 30 14:48:40.576906081 (599h19m40.447989136s) [noise=true , address=145.79.227.19:4078   , piece_id=IFNBG37H5W7XDCSRKF4RJ7RWW3RX4A3I337I3EKL57MKN5JVCQSA, seg_pos={0 0}, upload=525305]: Uploading piece...
Dec 30 14:48:40.577841381 (599h19m40.448924426s) [noise=true , address=77.90.31.11:4554     , piece_id=4EBP2YLPVM3Z5QVEA46WQFGEJ35BRH65KMJSE4GNOAR4AKQZBADA, seg_pos={0 0}, upload=525305]: Uploading piece...
Dec 30 14:48:40.578599776 (599h19m40.449682821s) [noise=true , address=145.79.227.13:4804   , piece_id=C46RM2I5BMO6BGQLYZDXHCKISVMLB2K4O6A6ZYH7Y5CT3GCBHK7A, seg_pos={0 0}, upload=525305]: Uploading piece...
Dec 30 14:48:40.578975962 (599h19m40.450059007s) [noise=true , address=145.79.227.50:4375   , piece_id=34Q4HWAON52CYHTIGYOG2B4HGUEKNMTXXYL7D73CSMSMMHV66YAA, seg_pos={0 0}, upload=525305]: Uploading piece...
Dec 30 14:48:40.580180947 (599h19m40.451263992s) [noise=true , address=77.90.31.14:4662     , piece_id=WERCE7Y6SGICRF2TYY2MWMD3WHLINNPTWBCFLGTJ6CNO4UI7UROA, seg_pos={0 0}, upload=525305]: Uploading piece...
Dec 30 14:48:40.580731725 (599h19m40.45181476s) [noise=true , address=145.79.227.32:4835   , piece_id=6BRQDB7BXVRMKPLIR7Q55HFMOD7CKRX7DEKCTNPPRVCYYUO7VT4Q, seg_pos={0 0}, upload=525305]: Uploading piece...
Dec 30 14:48:40.581876154 (599h19m40.452959199s) [noise=true , address=145.79.227.25:4619   , piece_id=CBIIOJVZSW6CMZPQG2U2VMOHJUXRX64LX2SBUUKSAC6M4GNZGOCQ, seg_pos={0 0}, upload=525305]: Uploading piece...
Dec 30 14:48:40.582754007 (599h19m40.453837042s) [noise=true , address=145.79.227.14:4839   , piece_id=E4XWU3PLKMQSAJQ34KM3PKNOY54URS5VDSQ5EMF42ZXEKPJDDTEA, seg_pos={0 0}, upload=525305]: Uploading piece...
Dec 30 14:48:40.584102364 (599h19m40.455185409s) [noise=true , address=145.79.227.36:4517   , piece_id=XKKMY27ER336G5ZY7RAN3RCFHSM62AVXK6BJZ7NXBPGZQRHZ4XJA, seg_pos={0 0}, upload=525305]: Uploading piece...
Dec 30 14:48:40.585286929 (599h19m40.456369964s) [noise=true , address=45.91.116.8:4045     , piece_id=FWTTV2XQAISSR2OTZCDBCKQWFKV6SW5IS43LSVHA7LUVYLEECKQQ, seg_pos={0 0}, upload=525305]: Uploading piece...
Dec 30 14:48:42.288325403 (599h19m42.159408438s) [upload=525305]: Next returned nil segment
Dec 30 14:48:42.288346453 (599h19m42.159429488s) [upload=525305]: Waiting for error group managing segments...
Dec 30 14:48:42.569043631 (599h19m42.440126666s) [noise=true , address=145.79.227.50:4375   , piece_id=34Q4HWAON52CYHTIGYOG2B4HGUEKNMTXXYL7D73CSMSMMHV66YAA, seg_pos={0 0}, upload=525305]: Done uploading piece. err: <nil>
Dec 30 14:48:42.570015721 (599h19m42.441098766s) [noise=true , address=145.79.227.13:4804   , piece_id=C46RM2I5BMO6BGQLYZDXHCKISVMLB2K4O6A6ZYH7Y5CT3GCBHK7A, seg_pos={0 0}, upload=525305]: Done uploading piece. err: <nil>
Dec 30 14:48:42.571243579 (599h19m42.442326624s) [noise=true , address=145.79.227.14:4839   , piece_id=E4XWU3PLKMQSAJQ34KM3PKNOY54URS5VDSQ5EMF42ZXEKPJDDTEA, seg_pos={0 0}, upload=525305]: Done uploading piece. err: <nil>
Dec 30 14:48:42.573200706 (599h19m42.444283741s) [noise=true , address=145.79.227.19:4078   , piece_id=IFNBG37H5W7XDCSRKF4RJ7RWW3RX4A3I337I3EKL57MKN5JVCQSA, seg_pos={0 0}, upload=525305]: Done uploading piece. err: <nil>
Dec 30 14:48:42.587901818 (599h19m42.458984853s) [noise=true , address=145.79.227.3:4149    , piece_id=BGSRACZLQKJOKG3IDSHVKM5SGRQ4SNTAKWZOK2LLMVNBOJ5KVG5A, seg_pos={0 0}, upload=525305]: Done uploading piece. err: <nil>
Dec 30 14:48:42.594804453 (599h19m42.465887488s) [noise=true , address=145.79.227.36:4517   , piece_id=XKKMY27ER336G5ZY7RAN3RCFHSM62AVXK6BJZ7NXBPGZQRHZ4XJA, seg_pos={0 0}, upload=525305]: Done uploading piece. err: <nil>
Dec 30 14:48:42.633356592 (599h19m42.504439627s) [noise=true , address=145.79.227.41:4087   , piece_id=GC7NQBAEZUVF7MUEBE6FRGTVM7YAHXGZTJV72WLVCL3ZOXES52CA, seg_pos={0 0}, upload=525305]: Done uploading piece. err: <nil>
Dec 30 14:48:42.648450473 (599h19m42.519533518s) [noise=true , address=145.79.227.17:4970   , piece_id=OJ5JPYFNHMDVGKSCSC4EHMINHI2BJDXYLOSUW2PYH2A647YVG7OQ, seg_pos={0 0}, upload=525305]: Done uploading piece. err: <nil>
Dec 30 14:48:42.649546449 (599h19m42.520629484s) [noise=true , address=145.79.227.25:4619   , piece_id=CBIIOJVZSW6CMZPQG2U2VMOHJUXRX64LX2SBUUKSAC6M4GNZGOCQ, seg_pos={0 0}, upload=525305]: Done uploading piece. err: <nil>
Dec 30 14:48:42.651815119 (599h19m42.522898164s) [noise=true , address=145.79.227.4:4185    , piece_id=AM2ET22HFDB5NS53OA46W2CDSQY5T7SIAGAK36BYPY2UGWDCIUQQ, seg_pos={0 0}, upload=525305]: Done uploading piece. err: <nil>
Dec 30 14:48:42.743152229 (599h19m42.614235264s) [noise=true , address=145.79.227.32:4835   , piece_id=6BRQDB7BXVRMKPLIR7Q55HFMOD7CKRX7DEKCTNPPRVCYYUO7VT4Q, seg_pos={0 0}, upload=525305]: Done uploading piece. err: <nil>
Dec 30 14:48:42.770672602 (599h19m42.641755637s) [noise=true , address=77.90.31.11:4554     , piece_id=4EBP2YLPVM3Z5QVEA46WQFGEJ35BRH65KMJSE4GNOAR4AKQZBADA, seg_pos={0 0}, upload=525305]: Done uploading piece. err: <nil>
Dec 30 14:48:42.908698433 (599h19m42.779781468s) [noise=true , address=45.91.116.5:4299     , piece_id=QBPAKBITBTU66GRCHIU47SNJ46L5DUAPA74NLYEFG7HD6N42QWEQ, seg_pos={0 0}, upload=525305]: Done uploading piece. err: <nil>
Dec 30 14:48:42.908716678 (599h19m42.779799713s) [seg_pos={0 0}, upload=525305]: Segment reached optimal threshold of 13 pieces.
Dec 30 14:48:42.908982754 (599h19m42.780065789s) [noise=true , address=77.90.31.15:4700     , piece_id=5CB3GBBASEB34KIVBKULFLJGMTF4M5B2OXDVPZ4VM5NU5UU4TNDQ, seg_pos={0 0}, upload=525305]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: write tcp 188.166.225.170:43598->77.90.31.15:4700: use of closed network connection; write tcp 188.166.225.170:43598->77.90.31.15:4700: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43598->77.90.31.15:4700: use of closed network connection
Dec 30 14:48:42.908990339 (599h19m42.780073374s) [noise=true , address=77.90.31.15:4700     , piece_id=5CB3GBBASEB34KIVBKULFLJGMTF4M5B2OXDVPZ4VM5NU5UU4TNDQ, seg_pos={0 0}, upload=525305]: Long tail cancellation due to optimal threshold
Dec 30 14:48:42.909210759 (599h19m42.780293794s) [noise=true , address=45.91.116.8:4045     , piece_id=FWTTV2XQAISSR2OTZCDBCKQWFKV6SW5IS43LSVHA7LUVYLEECKQQ, seg_pos={0 0}, upload=525305]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF
Dec 30 14:48:42.909225276 (599h19m42.780308311s) [noise=true , address=45.91.116.8:4045     , piece_id=FWTTV2XQAISSR2OTZCDBCKQWFKV6SW5IS43LSVHA7LUVYLEECKQQ, seg_pos={0 0}, upload=525305]: Long tail cancellation due to optimal threshold
Dec 30 14:48:42.909273809 (599h19m42.780356844s) [noise=true , address=77.90.31.14:4662     , piece_id=WERCE7Y6SGICRF2TYY2MWMD3WHLINNPTWBCFLGTJ6CNO4UI7UROA, seg_pos={0 0}, upload=525305]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF
Dec 30 14:48:42.909281794 (599h19m42.780364829s) [noise=true , address=77.90.31.14:4662     , piece_id=WERCE7Y6SGICRF2TYY2MWMD3WHLINNPTWBCFLGTJ6CNO4UI7UROA, seg_pos={0 0}, upload=525305]: Long tail cancellation due to optimal threshold
Dec 30 14:48:42.909304437 (599h19m42.780387472s) [seg_pos={0 0}, upload=525305]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 14:48:42.909372617 (599h19m42.780455652s) [upload=525305]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 14:48:42.909377847 (599h19m42.780460882s) [upload=525305]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 14:48:43.222464539 (599h19m43.093547574s) [upload=525305]: Done uploading segments...
Dec 30 14:48:43.222482663 (599h19m43.093565698s) [upload=525305]: Upload finished. err: <nil>
Duration: 30.00327243s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)
antiblip: 
4 out of 60 is failed (threshold=3) [X.X.X........................................X..............]
duration: 
15.386289253s
trace: 
https://jaeger.dev.storj.io/trace/71938bc9cbb32ff3
FAIL (2025-12-30 14:48:53 UTC) 
orbiter warning: context deadline exceeded
Duration: 15.386336362s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
47 out of 60 is failed (threshold=6) [XXXXXXXX.XXXXXXXXXXXXXXX.X..XX.XXXX.XXXXXXX.X..XX.X...XXXXXX]
duration: 
5.001193892s
trace: 
https://jaeger.dev.storj.io/trace/155652c818373dda
FAIL (2025-12-30 14:48:38 UTC) 
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001212015s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
15 out of 60 is failed (threshold=3) [X.XXX.....X......X......X........X.X.X..X..X.....XXX........]
duration: 
18.746411246s
trace: 
https://jaeger.dev.storj.io/trace/41ecdbaa0176ef93
FAIL (2025-12-30 14:48:36 UTC) 
orbiter warning: context deadline exceeded
Duration: 18.746439971s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)
antiblip: 
1 out of 60 is failed (threshold=3) [..............................................X.............]
duration: 
5.376629175s
trace: 
https://jaeger.dev.storj.io/trace/414189176d03adca
FAIL (2025-12-30 14:48:33 UTC) 
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 5.376659052s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [................................X...........................]
duration: 
57.356842142s
trace: 
https://jaeger.dev.storj.io/trace/6a6762cfd537d983
FAIL (2025-12-30 14:48:08 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
JYHN47QHRILZESBC2RFMKJV5GL3P2MDYCEIC6OOCIMDFXHJA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.155565546 (598h55m6.026648581s) [noise=true , address=77.90.31.12:4603     , piece_id=VHIHG4OLDWDIVEUEYHK4XT744P3MW7E7KNUYNFZTQ22YX4RPARDA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.156109141 (598h55m6.027192186s) [noise=true , address=145.79.227.40:4072   , piece_id=A5BM54GXYHHDXLKFUYYR4EOCH5VGTJCAG62R4FCHGREBAFYRLKVQ, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.156505756 (598h55m6.027588801s) [noise=true , address=145.79.227.44:4208   , piece_id=PVBUF6UBX6Y3TEVO44WFXZIOXZTYZOJ4R5GPKRKUFTNOV7RUGKOQ, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.156892222 (598h55m6.027975267s) [noise=true , address=77.90.31.11:4575     , piece_id=FCTSMJOTE6GESX2KMZIFSQQ2KVPO5KLJY4UAF2LCTCFNCPB7KAKA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.157282115 (598h55m6.02836516s) [noise=true , address=64.49.13.169:4708    , piece_id=UIEZEM3SBZAU5ZKRU6PTYXHKTGEJKUJJ7ZGKS6VI6Q76QCUPM2UA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.157661989 (598h55m6.028745024s) [noise=true , address=77.90.31.21:4901     , piece_id=7QGD4I6XEQL6WRP7VFQDA53CVPB4YUANQ44II4XMCNZXD4XUUXNQ, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.158076959 (598h55m6.029160004s) [noise=true , address=77.90.31.19:4873     , piece_id=MPBWXZI4ECW54FXCK5GZLFMXJV6XVQHQFAGX6NRSUZ2IQJVVN2AQ, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.158174844 (598h55m6.029257889s) [noise=true , address=77.90.31.14:4680     , piece_id=UYYWRJXI54VNJCGACTIH3EOETSSZLMGBOEDXKY53JBQ4SYTHAQJQ, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.158483282 (598h55m6.029566327s) [noise=true , address=145.79.227.29:4958   , piece_id=HYFYCKV2LPUBYZPJ6XSBJ34OHZTCRYFERS3O6J5KADS2POL27CSA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.158804814 (598h55m6.029887859s) [noise=true , address=145.79.227.8:4343    , piece_id=CXRMAPWCX5EI4JHWJ5M6Z3QVIHZB3OBD5AZPOZN5TMKS2QDSCACQ, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.159532480 (598h55m6.030615515s) [noise=true , address=145.79.227.49:4363   , piece_id=UJBKNKMCTCEXRVTN3E77Y4DHN2LY7TFHWPOTEO7753UGFXBYYG3A, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.160163792 (598h55m6.031246837s) [noise=true , address=145.79.227.30:4768   , piece_id=GYOQQE5HMHPQCZQMLXELGSFA5ESRVFQNHHHVF25TLDH4FHCWE44A, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.160862814 (598h55m6.031945849s) [noise=true , address=145.79.227.18:4016   , piece_id=C66NGRXYHUPALVM3NM2FSHSGKX6GL7IMTUGXFIYCEJY6X4Z5PKLA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.161559340 (598h55m6.032642375s) [noise=true , address=145.79.227.34:4925   , piece_id=OAJG6P3LWWKQKNN3FXCHEY4LT4SGUC3PXYGEYY26J5TZ3XD4QBKA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.162131689 (598h55m6.033214724s) [noise=true , address=77.90.31.15:4710     , piece_id=U4QZDJXGNFNEVGYOIXCLKI3EW5EZYJWTE5KH4CVAF7DVFXXSTKTA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:07.617768952 (598h55m7.488851997s) [upload=524959]: Next returned nil segment
Dec 30 14:24:07.617782789 (598h55m7.488865824s) [upload=524959]: Waiting for error group managing segments...
Dec 30 14:24:08.019443601 (598h55m7.890526646s) [noise=true , address=145.79.227.29:4958   , piece_id=HYFYCKV2LPUBYZPJ6XSBJ34OHZTCRYFERS3O6J5KADS2POL27CSA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.054613990 (598h55m7.925697025s) [noise=true , address=145.79.227.18:4016   , piece_id=C66NGRXYHUPALVM3NM2FSHSGKX6GL7IMTUGXFIYCEJY6X4Z5PKLA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.096469250 (598h55m7.967552295s) [noise=true , address=145.79.227.8:4343    , piece_id=CXRMAPWCX5EI4JHWJ5M6Z3QVIHZB3OBD5AZPOZN5TMKS2QDSCACQ, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.342930639 (598h55m8.214013674s) [noise=true , address=77.90.31.12:4603     , piece_id=VHIHG4OLDWDIVEUEYHK4XT744P3MW7E7KNUYNFZTQ22YX4RPARDA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.344873098 (598h55m8.215956133s) [noise=true , address=77.90.31.19:4873     , piece_id=MPBWXZI4ECW54FXCK5GZLFMXJV6XVQHQFAGX6NRSUZ2IQJVVN2AQ, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.359458851 (598h55m8.230541887s) [noise=true , address=77.90.31.14:4680     , piece_id=UYYWRJXI54VNJCGACTIH3EOETSSZLMGBOEDXKY53JBQ4SYTHAQJQ, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.446410890 (598h55m8.317493925s) [noise=true , address=77.90.31.21:4901     , piece_id=7QGD4I6XEQL6WRP7VFQDA53CVPB4YUANQ44II4XMCNZXD4XUUXNQ, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.950565858 (598h55m8.821648893s) [noise=true , address=145.79.227.33:4878   , piece_id=2GGG474UM3S65BFMDV2MFWBUCYFM62Y4MWHQADZ62WHGSLBF67LA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.963965914 (598h55m8.835048949s) [noise=true , address=145.79.227.30:4768   , piece_id=GYOQQE5HMHPQCZQMLXELGSFA5ESRVFQNHHHVF25TLDH4FHCWE44A, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.987544621 (598h55m8.858627666s) [noise=true , address=145.79.227.34:4925   , piece_id=OAJG6P3LWWKQKNN3FXCHEY4LT4SGUC3PXYGEYY26J5TZ3XD4QBKA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.997751366 (598h55m8.868834411s) [noise=true , address=145.79.227.44:4208   , piece_id=PVBUF6UBX6Y3TEVO44WFXZIOXZTYZOJ4R5GPKRKUFTNOV7RUGKOQ, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:09.013743367 (598h55m8.884826412s) [noise=true , address=145.79.227.40:4072   , piece_id=A5BM54GXYHHDXLKFUYYR4EOCH5VGTJCAG62R4FCHGREBAFYRLKVQ, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:09.042854289 (598h55m8.913937334s) [noise=true , address=145.79.227.49:4363   , piece_id=UJBKNKMCTCEXRVTN3E77Y4DHN2LY7TFHWPOTEO7753UGFXBYYG3A, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:09.139174176 (598h55m9.010257221s) [noise=true , address=145.79.227.43:4174   , piece_id=NBFMEM5UTODEKLDJ7DESZVSWMYZSL3H2H4FPQE54II6SKUOMPYEA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:09.139199304 (598h55m9.010282349s) [seg_pos={0 0}, upload=524959]: Segment reached optimal threshold of 14 pieces.
Dec 30 14:24:09.139565892 (598h55m9.010648937s) [noise=true , address=77.90.31.13:4633     , piece_id=KCJXJYHN47QHRILZESBC2RFMKJV5GL3P2MDYCEIC6OOCIMDFXHJA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qXhyMbfXELW1Vg5z9NpqFX1fm3v2pGEaA56qSc8kHefPNNNYs): protocol: expected piece hash; context canceled; EOF
Dec 30 14:24:09.139575490 (598h55m9.010658535s) [noise=true , address=77.90.31.13:4633     , piece_id=KCJXJYHN47QHRILZESBC2RFMKJV5GL3P2MDYCEIC6OOCIMDFXHJA, seg_pos={0 0}, upload=524959]: Long tail cancellation due to optimal threshold
Dec 30 14:24:09.140024375 (598h55m9.01110742s) [noise=true , address=77.90.31.15:4710     , piece_id=U4QZDJXGNFNEVGYOIXCLKI3EW5EZYJWTE5KH4CVAF7DVFXXSTKTA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iHBz7GEqPcdQPQfzTKm3dLkUNT279gbgEh8opdukSjubKrz7X): protocol: expected piece hash; context canceled; EOF
Dec 30 14:24:09.140034804 (598h55m9.01111784s) [noise=true , address=77.90.31.15:4710     , piece_id=U4QZDJXGNFNEVGYOIXCLKI3EW5EZYJWTE5KH4CVAF7DVFXXSTKTA, seg_pos={0 0}, upload=524959]: Long tail cancellation due to optimal threshold
Dec 30 14:24:09.140346318 (598h55m9.011429363s) [noise=true , address=77.90.31.11:4575     , piece_id=FCTSMJOTE6GESX2KMZIFSQQ2KVPO5KLJY4UAF2LCTCFNCPB7KAKA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc): protocol: read tcp 188.166.225.170:58170->77.90.31.11:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 30 14:24:09.140355967 (598h55m9.011439012s) [noise=true , address=77.90.31.11:4575     , piece_id=FCTSMJOTE6GESX2KMZIFSQQ2KVPO5KLJY4UAF2LCTCFNCPB7KAKA, seg_pos={0 0}, upload=524959]: Long tail cancellation due to optimal threshold
Dec 30 14:24:09.140523655 (598h55m9.0116067s) [noise=true , address=64.49.13.169:4708    , piece_id=UIEZEM3SBZAU5ZKRU6PTYXHKTGEJKUJJ7ZGKS6VI6Q76QCUPM2UA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:43936->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:43936->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43936->64.49.13.169:4708: use of closed network connection
Dec 30 14:24:09.140741952 (598h55m9.011824997s) [noise=true , address=64.49.13.169:4708    , piece_id=UIEZEM3SBZAU5ZKRU6PTYXHKTGEJKUJJ7ZGKS6VI6Q76QCUPM2UA, seg_pos={0 0}, upload=524959]: Long tail cancellation due to optimal threshold
Dec 30 14:24:09.140766057 (598h55m9.011849102s) [seg_pos={0 0}, upload=524959]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 14:24:09.140967651 (598h55m9.012050686s) [upload=524959]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 14:24:09.141080235 (598h55m9.01216328s) [upload=524959]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 14:24:09.447704404 (598h55m9.318787449s) [upload=524959]: Done uploading segments...
Dec 30 14:24:09.447727117 (598h55m9.318810162s) [upload=524959]: Upload finished. err: <nil>
Duration: 57.3568672s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)
antiblip: 
4 out of 60 is failed (threshold=3) [.X.X.X........................................X.............]
duration: 
14.606804585s
trace: 
https://jaeger.dev.storj.io/trace/71938bc9cbb32ff3
FAIL (2025-12-30 14:47:52 UTC) 
orbiter warning: context deadline exceeded
Duration: 14.606830503s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)
antiblip: 
1 out of 60 is failed (threshold=3) [...............................................X............]
duration: 
10.281446683s
trace: 
https://jaeger.dev.storj.io/trace/414189176d03adca
FAIL (2025-12-30 14:47:46 UTC) 
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 10.281471141s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
15 out of 60 is failed (threshold=3) [...X.....X...X....X......X.......X..X....X..X...X..XXX..XX..]
duration: 
5.960818257s
trace: 
https://jaeger.dev.storj.io/trace/36dad44f99247e7a
FAIL (2025-12-30 14:47:44 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ments...
Dec 30 14:45:48.177480448 (599h16m48.048563483s) [upload=525269]: Got next segment. Inline: false
Dec 30 14:45:48.177643258 (599h16m48.048726313s) [upload=525269]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 14:45:48.177650301 (599h16m48.048733336s) [upload=525269]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 14:45:48.422853313 (599h16m48.293936358s) [seg_pos={0 0}, upload=525269]: Begin upload segment...
Dec 30 14:45:48.422920802 (599h16m48.294003837s) [seg_pos={0 0}, upload=525269]: Done begin upload segment.
Dec 30 14:45:48.422950208 (599h16m48.294033243s) [noise=true , address=45.91.116.14:4189    , piece_id=7HXFSRSYQ256METQQC7B7SEPSGMRBDWUBS24GBKYP5HHR6EF7ANQ, seg_pos={0 0}, upload=525269]: Uploading piece...
Dec 30 14:45:48.423024379 (599h16m48.294107414s) [noise=true , address=145.79.227.38:4977   , piece_id=N7DN5Y236D5CGVEVNALRIKLZ3VZHMCWI4LY5OMEOICZK2RRFQJVQ, seg_pos={0 0}, upload=525269]: Uploading piece...
Dec 30 14:45:48.423775730 (599h16m48.294858775s) [noise=true , address=77.90.31.13:4629     , piece_id=ZIU5QZFTCYDJ763ZQ3IFORA7LB5QXGNM2AQTXAQBTLTIFKUE4HLQ, seg_pos={0 0}, upload=525269]: Uploading piece...
Dec 30 14:45:48.424521800 (599h16m48.295604845s) [noise=true , address=145.79.227.20:4439   , piece_id=44EYLNDP2MHNVH3V6BB4X2PAA4FLL4TP66RUQLBFGONP6CRAGYGA, seg_pos={0 0}, upload=525269]: Uploading piece...
Dec 30 14:45:48.424770584 (599h16m48.295853629s) [noise=true , address=45.91.116.6:4975     , piece_id=5FDXIGSVSWXKEHCHY2NPJVYYUVUTLQ6HSFP4RY6GQWHANQRYV2DA, seg_pos={0 0}, upload=525269]: Uploading piece...
Dec 30 14:45:48.424998518 (599h16m48.296081553s) [noise=true , address=45.91.116.10:4084    , piece_id=URR4NNW4YWY46MDEJHIMMYPFWGLEKGLC3RXMMLBXTX4COTF6FJCA, seg_pos={0 0}, upload=525269]: Uploading piece...
Dec 30 14:45:48.425444368 (599h16m48.296527413s) [noise=true , address=145.79.227.52:4446   , piece_id=5GLI3JQW3HGMSC5AHD3PDWU42TQT4SXYKP3FXZ5WH3Z4NGS7NJVQ, seg_pos={0 0}, upload=525269]: Uploading piece...
Dec 30 14:45:48.425577942 (599h16m48.296660987s) [noise=true , address=145.79.227.50:4375   , piece_id=W4QDF5DQLGG4C74U3YIHMU3PQNNNU67UOE2EYM3CO6TANR7DWFQQ, seg_pos={0 0}, upload=525269]: Uploading piece...
Dec 30 14:45:48.426069197 (599h16m48.297152232s) [noise=true , address=145.79.227.17:4970   , piece_id=HG7GSWPRJD6ALYZ77FNGJVWRUSUIDROX2VQ3EWYYDBY2X3UKBN6A, seg_pos={0 0}, upload=525269]: Uploading piece...
Dec 30 14:45:48.426162094 (599h16m48.297245139s) [noise=true , address=145.79.227.39:4014   , piece_id=WRKYHSEN6KGZMT474SPO4WFJ33YESQUJPGU6PYWFZSZC5QPGQC4Q, seg_pos={0 0}, upload=525269]: Uploading piece...
Dec 30 14:45:48.426470621 (599h16m48.297553666s) [noise=true , address=77.90.31.19:4844     , piece_id=FDSRTLAK6TTUXOVCOI4MIMCCBBPNCSUEDJAA6R4FXR7ZUGPH2RZQ, seg_pos={0 0}, upload=525269]: Uploading piece...
Dec 30 14:45:48.426717070 (599h16m48.297800106s) [noise=true , address=145.79.227.36:4518   , piece_id=SPCRCKPF5DSZZUGTJOTOZWV56WFT6MQUQUIMTIPN2DRPUIEHXDYQ, seg_pos={0 0}, upload=525269]: Uploading piece...
Dec 30 14:45:48.427027211 (599h16m48.298110256s) [noise=true , address=145.79.227.6:4257    , piece_id=BSSC6DFKQXZVQKOZNJOP4SOVCY22YTCQ6RMSIV7W2RHUSTSYAYCQ, seg_pos={0 0}, upload=525269]: Uploading piece...
Dec 30 14:45:48.427359975 (599h16m48.29844302s) [noise=true , address=77.90.31.14:4664     , piece_id=WUDQG643OQXDXXW4UYYZQJ2DQDPRIP5K4RA77CKDD4QT4TAKX4HQ, seg_pos={0 0}, upload=525269]: Uploading piece...
Dec 30 14:45:48.428091438 (599h16m48.299174473s) [noise=true , address=145.79.227.3:4149    , piece_id=4FFXO7MYQ7U5MLDAX464HVEXOUNGJI3MVOYHT5KDTD355WLVHJTA, seg_pos={0 0}, upload=525269]: Uploading piece...
Dec 30 14:45:48.428498633 (599h16m48.299581678s) [noise=true , address=145.79.227.8:4329    , piece_id=YKBRJOTUHU6NEUU7QEDRWVGB7YXL2BIZPN57KB56ILUCRR7SVXOQ, seg_pos={0 0}, upload=525269]: Uploading piece...
Dec 30 14:45:50.128610877 (599h16m49.999693922s) [upload=525269]: Next returned nil segment
Dec 30 14:45:50.128626947 (599h16m49.999709992s) [upload=525269]: Waiting for error group managing segments...
Dec 30 14:45:50.408609917 (599h16m50.279692952s) [noise=true , address=145.79.227.17:4970   , piece_id=HG7GSWPRJD6ALYZ77FNGJVWRUSUIDROX2VQ3EWYYDBY2X3UKBN6A, seg_pos={0 0}, upload=525269]: Done uploading piece. err: <nil>
Dec 30 14:45:50.421485384 (599h16m50.292568429s) [noise=true , address=145.79.227.20:4439   , piece_id=44EYLNDP2MHNVH3V6BB4X2PAA4FLL4TP66RUQLBFGONP6CRAGYGA, seg_pos={0 0}, upload=525269]: Done uploading piece. err: <nil>
Dec 30 14:45:50.421927636 (599h16m50.293010671s) [noise=true , address=145.79.227.39:4014   , piece_id=WRKYHSEN6KGZMT474SPO4WFJ33YESQUJPGU6PYWFZSZC5QPGQC4Q, seg_pos={0 0}, upload=525269]: Done uploading piece. err: <nil>
Dec 30 14:45:50.434527668 (599h16m50.305610703s) [noise=true , address=145.79.227.3:4149    , piece_id=4FFXO7MYQ7U5MLDAX464HVEXOUNGJI3MVOYHT5KDTD355WLVHJTA, seg_pos={0 0}, upload=525269]: Done uploading piece. err: <nil>
Dec 30 14:45:50.437145943 (599h16m50.308228978s) [noise=true , address=145.79.227.38:4977   , piece_id=N7DN5Y236D5CGVEVNALRIKLZ3VZHMCWI4LY5OMEOICZK2RRFQJVQ, seg_pos={0 0}, upload=525269]: Done uploading piece. err: <nil>
Dec 30 14:45:50.437361825 (599h16m50.30844487s) [noise=true , address=145.79.227.36:4518   , piece_id=SPCRCKPF5DSZZUGTJOTOZWV56WFT6MQUQUIMTIPN2DRPUIEHXDYQ, seg_pos={0 0}, upload=525269]: Done uploading piece. err: <nil>
Dec 30 14:45:50.507310808 (599h16m50.378393853s) [noise=true , address=145.79.227.52:4446   , piece_id=5GLI3JQW3HGMSC5AHD3PDWU42TQT4SXYKP3FXZ5WH3Z4NGS7NJVQ, seg_pos={0 0}, upload=525269]: Done uploading piece. err: <nil>
Dec 30 14:45:50.522382497 (599h16m50.393465542s) [noise=true , address=145.79.227.8:4329    , piece_id=YKBRJOTUHU6NEUU7QEDRWVGB7YXL2BIZPN57KB56ILUCRR7SVXOQ, seg_pos={0 0}, upload=525269]: Done uploading piece. err: <nil>
Dec 30 14:45:50.523916387 (599h16m50.394999422s) [noise=true , address=145.79.227.6:4257    , piece_id=BSSC6DFKQXZVQKOZNJOP4SOVCY22YTCQ6RMSIV7W2RHUSTSYAYCQ, seg_pos={0 0}, upload=525269]: Done uploading piece. err: <nil>
Dec 30 14:45:50.548914517 (599h16m50.419997552s) [noise=true , address=145.79.227.50:4375   , piece_id=W4QDF5DQLGG4C74U3YIHMU3PQNNNU67UOE2EYM3CO6TANR7DWFQQ, seg_pos={0 0}, upload=525269]: Done uploading piece. err: <nil>
Dec 30 14:45:50.822818631 (599h16m50.693901686s) [noise=true , address=45.91.116.6:4975     , piece_id=5FDXIGSVSWXKEHCHY2NPJVYYUVUTLQ6HSFP4RY6GQWHANQRYV2DA, seg_pos={0 0}, upload=525269]: Done uploading piece. err: <nil>
Dec 30 14:45:51.229364634 (599h16m51.100447669s) [noise=true , address=45.91.116.10:4084    , piece_id=URR4NNW4YWY46MDEJHIMMYPFWGLEKGLC3RXMMLBXTX4COTF6FJCA, seg_pos={0 0}, upload=525269]: Done uploading piece. err: <nil>
Dec 30 14:45:51.373194307 (599h16m51.244277342s) [noise=true , address=77.90.31.13:4629     , piece_id=ZIU5QZFTCYDJ763ZQ3IFORA7LB5QXGNM2AQTXAQBTLTIFKUE4HLQ, seg_pos={0 0}, upload=525269]: Done uploading piece. err: <nil>
Dec 30 14:45:51.373218744 (599h16m51.244301789s) [seg_pos={0 0}, upload=525269]: Segment reached optimal threshold of 13 pieces.
Dec 30 14:45:51.373443942 (599h16m51.244526987s) [noise=true , address=45.91.116.14:4189    , piece_id=7HXFSRSYQ256METQQC7B7SEPSGMRBDWUBS24GBKYP5HHR6EF7ANQ, seg_pos={0 0}, upload=525269]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:57656->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:57656->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57656->45.91.116.14:4189: use of closed network connection
Dec 30 14:45:51.373454382 (599h16m51.244537427s) [noise=true , address=45.91.116.14:4189    , piece_id=7HXFSRSYQ256METQQC7B7SEPSGMRBDWUBS24GBKYP5HHR6EF7ANQ, seg_pos={0 0}, upload=525269]: Long tail cancellation due to optimal threshold
Dec 30 14:45:51.373498757 (599h16m51.244581802s) [noise=true , address=77.90.31.14:4664     , piece_id=WUDQG643OQXDXXW4UYYZQJ2DQDPRIP5K4RA77CKDD4QT4TAKX4HQ, seg_pos={0 0}, upload=525269]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: write tcp 188.166.225.170:54652->77.90.31.14:4664: use of closed network connection; write tcp 188.166.225.170:54652->77.90.31.14:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54652->77.90.31.14:4664: use of closed network connection
Dec 30 14:45:51.373515739 (599h16m51.244598784s) [noise=true , address=77.90.31.14:4664     , piece_id=WUDQG643OQXDXXW4UYYZQJ2DQDPRIP5K4RA77CKDD4QT4TAKX4HQ, seg_pos={0 0}, upload=525269]: Long tail cancellation due to optimal threshold
Dec 30 14:45:51.373854494 (599h16m51.244937549s) [noise=true , address=77.90.31.19:4844     , piece_id=FDSRTLAK6TTUXOVCOI4MIMCCBBPNCSUEDJAA6R4FXR7ZUGPH2RZQ, seg_pos={0 0}, upload=525269]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF
Dec 30 14:45:51.373883971 (599h16m51.244967006s) [noise=true , address=77.90.31.19:4844     , piece_id=FDSRTLAK6TTUXOVCOI4MIMCCBBPNCSUEDJAA6R4FXR7ZUGPH2RZQ, seg_pos={0 0}, upload=525269]: Long tail cancellation due to optimal threshold
Dec 30 14:45:51.373921853 (599h16m51.245004898s) [seg_pos={0 0}, upload=525269]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 14:45:51.374024318 (599h16m51.245107363s) [upload=525269]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 14:45:51.374033335 (599h16m51.24511638s) [upload=525269]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 14:45:51.666271069 (599h16m51.537354114s) [upload=525269]: Done uploading segments...
Dec 30 14:45:51.666295586 (599h16m51.537378651s) [upload=525269]: Upload finished. err: <nil>
Duration: 5.960835651s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
16 out of 60 is failed (threshold=3) [XX.XXX.....X......X......X........X.X.X..X..X.....XXX.......]
duration: 
16.711992537s
trace: 
https://jaeger.dev.storj.io/trace/41ecdbaa0176ef93
FAIL (2025-12-30 14:47:34 UTC) 
orbiter warning: context deadline exceeded
Duration: 16.712014249s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
47 out of 60 is failed (threshold=6) [XXXXXXXXX.XXXXXXXXXXXXXXX.X..XX.XXXX.XXXXXXX.X..XX.X...XXXXX]
duration: 
5.000908447s
trace: 
https://jaeger.dev.storj.io/trace/4a19c82409dc6eb9
FAIL (2025-12-30 14:47:29 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000925399s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)
antiblip: 
1 out of 60 is failed (threshold=3) [................................................X...........]
duration: 
10.850471067s
trace: 
https://jaeger.dev.storj.io/trace/414189176d03adca
FAIL (2025-12-30 14:47:25 UTC) 
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 10.850496454s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [.................................X..........................]
duration: 
38.875767743s
trace: 
https://jaeger.dev.storj.io/trace/6a6762cfd537d983
FAIL (2025-12-30 14:47:01 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
JYHN47QHRILZESBC2RFMKJV5GL3P2MDYCEIC6OOCIMDFXHJA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.155565546 (598h55m6.026648581s) [noise=true , address=77.90.31.12:4603     , piece_id=VHIHG4OLDWDIVEUEYHK4XT744P3MW7E7KNUYNFZTQ22YX4RPARDA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.156109141 (598h55m6.027192186s) [noise=true , address=145.79.227.40:4072   , piece_id=A5BM54GXYHHDXLKFUYYR4EOCH5VGTJCAG62R4FCHGREBAFYRLKVQ, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.156505756 (598h55m6.027588801s) [noise=true , address=145.79.227.44:4208   , piece_id=PVBUF6UBX6Y3TEVO44WFXZIOXZTYZOJ4R5GPKRKUFTNOV7RUGKOQ, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.156892222 (598h55m6.027975267s) [noise=true , address=77.90.31.11:4575     , piece_id=FCTSMJOTE6GESX2KMZIFSQQ2KVPO5KLJY4UAF2LCTCFNCPB7KAKA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.157282115 (598h55m6.02836516s) [noise=true , address=64.49.13.169:4708    , piece_id=UIEZEM3SBZAU5ZKRU6PTYXHKTGEJKUJJ7ZGKS6VI6Q76QCUPM2UA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.157661989 (598h55m6.028745024s) [noise=true , address=77.90.31.21:4901     , piece_id=7QGD4I6XEQL6WRP7VFQDA53CVPB4YUANQ44II4XMCNZXD4XUUXNQ, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.158076959 (598h55m6.029160004s) [noise=true , address=77.90.31.19:4873     , piece_id=MPBWXZI4ECW54FXCK5GZLFMXJV6XVQHQFAGX6NRSUZ2IQJVVN2AQ, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.158174844 (598h55m6.029257889s) [noise=true , address=77.90.31.14:4680     , piece_id=UYYWRJXI54VNJCGACTIH3EOETSSZLMGBOEDXKY53JBQ4SYTHAQJQ, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.158483282 (598h55m6.029566327s) [noise=true , address=145.79.227.29:4958   , piece_id=HYFYCKV2LPUBYZPJ6XSBJ34OHZTCRYFERS3O6J5KADS2POL27CSA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.158804814 (598h55m6.029887859s) [noise=true , address=145.79.227.8:4343    , piece_id=CXRMAPWCX5EI4JHWJ5M6Z3QVIHZB3OBD5AZPOZN5TMKS2QDSCACQ, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.159532480 (598h55m6.030615515s) [noise=true , address=145.79.227.49:4363   , piece_id=UJBKNKMCTCEXRVTN3E77Y4DHN2LY7TFHWPOTEO7753UGFXBYYG3A, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.160163792 (598h55m6.031246837s) [noise=true , address=145.79.227.30:4768   , piece_id=GYOQQE5HMHPQCZQMLXELGSFA5ESRVFQNHHHVF25TLDH4FHCWE44A, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.160862814 (598h55m6.031945849s) [noise=true , address=145.79.227.18:4016   , piece_id=C66NGRXYHUPALVM3NM2FSHSGKX6GL7IMTUGXFIYCEJY6X4Z5PKLA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.161559340 (598h55m6.032642375s) [noise=true , address=145.79.227.34:4925   , piece_id=OAJG6P3LWWKQKNN3FXCHEY4LT4SGUC3PXYGEYY26J5TZ3XD4QBKA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.162131689 (598h55m6.033214724s) [noise=true , address=77.90.31.15:4710     , piece_id=U4QZDJXGNFNEVGYOIXCLKI3EW5EZYJWTE5KH4CVAF7DVFXXSTKTA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:07.617768952 (598h55m7.488851997s) [upload=524959]: Next returned nil segment
Dec 30 14:24:07.617782789 (598h55m7.488865824s) [upload=524959]: Waiting for error group managing segments...
Dec 30 14:24:08.019443601 (598h55m7.890526646s) [noise=true , address=145.79.227.29:4958   , piece_id=HYFYCKV2LPUBYZPJ6XSBJ34OHZTCRYFERS3O6J5KADS2POL27CSA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.054613990 (598h55m7.925697025s) [noise=true , address=145.79.227.18:4016   , piece_id=C66NGRXYHUPALVM3NM2FSHSGKX6GL7IMTUGXFIYCEJY6X4Z5PKLA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.096469250 (598h55m7.967552295s) [noise=true , address=145.79.227.8:4343    , piece_id=CXRMAPWCX5EI4JHWJ5M6Z3QVIHZB3OBD5AZPOZN5TMKS2QDSCACQ, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.342930639 (598h55m8.214013674s) [noise=true , address=77.90.31.12:4603     , piece_id=VHIHG4OLDWDIVEUEYHK4XT744P3MW7E7KNUYNFZTQ22YX4RPARDA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.344873098 (598h55m8.215956133s) [noise=true , address=77.90.31.19:4873     , piece_id=MPBWXZI4ECW54FXCK5GZLFMXJV6XVQHQFAGX6NRSUZ2IQJVVN2AQ, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.359458851 (598h55m8.230541887s) [noise=true , address=77.90.31.14:4680     , piece_id=UYYWRJXI54VNJCGACTIH3EOETSSZLMGBOEDXKY53JBQ4SYTHAQJQ, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.446410890 (598h55m8.317493925s) [noise=true , address=77.90.31.21:4901     , piece_id=7QGD4I6XEQL6WRP7VFQDA53CVPB4YUANQ44II4XMCNZXD4XUUXNQ, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.950565858 (598h55m8.821648893s) [noise=true , address=145.79.227.33:4878   , piece_id=2GGG474UM3S65BFMDV2MFWBUCYFM62Y4MWHQADZ62WHGSLBF67LA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.963965914 (598h55m8.835048949s) [noise=true , address=145.79.227.30:4768   , piece_id=GYOQQE5HMHPQCZQMLXELGSFA5ESRVFQNHHHVF25TLDH4FHCWE44A, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.987544621 (598h55m8.858627666s) [noise=true , address=145.79.227.34:4925   , piece_id=OAJG6P3LWWKQKNN3FXCHEY4LT4SGUC3PXYGEYY26J5TZ3XD4QBKA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.997751366 (598h55m8.868834411s) [noise=true , address=145.79.227.44:4208   , piece_id=PVBUF6UBX6Y3TEVO44WFXZIOXZTYZOJ4R5GPKRKUFTNOV7RUGKOQ, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:09.013743367 (598h55m8.884826412s) [noise=true , address=145.79.227.40:4072   , piece_id=A5BM54GXYHHDXLKFUYYR4EOCH5VGTJCAG62R4FCHGREBAFYRLKVQ, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:09.042854289 (598h55m8.913937334s) [noise=true , address=145.79.227.49:4363   , piece_id=UJBKNKMCTCEXRVTN3E77Y4DHN2LY7TFHWPOTEO7753UGFXBYYG3A, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:09.139174176 (598h55m9.010257221s) [noise=true , address=145.79.227.43:4174   , piece_id=NBFMEM5UTODEKLDJ7DESZVSWMYZSL3H2H4FPQE54II6SKUOMPYEA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:09.139199304 (598h55m9.010282349s) [seg_pos={0 0}, upload=524959]: Segment reached optimal threshold of 14 pieces.
Dec 30 14:24:09.139565892 (598h55m9.010648937s) [noise=true , address=77.90.31.13:4633     , piece_id=KCJXJYHN47QHRILZESBC2RFMKJV5GL3P2MDYCEIC6OOCIMDFXHJA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qXhyMbfXELW1Vg5z9NpqFX1fm3v2pGEaA56qSc8kHefPNNNYs): protocol: expected piece hash; context canceled; EOF
Dec 30 14:24:09.139575490 (598h55m9.010658535s) [noise=true , address=77.90.31.13:4633     , piece_id=KCJXJYHN47QHRILZESBC2RFMKJV5GL3P2MDYCEIC6OOCIMDFXHJA, seg_pos={0 0}, upload=524959]: Long tail cancellation due to optimal threshold
Dec 30 14:24:09.140024375 (598h55m9.01110742s) [noise=true , address=77.90.31.15:4710     , piece_id=U4QZDJXGNFNEVGYOIXCLKI3EW5EZYJWTE5KH4CVAF7DVFXXSTKTA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iHBz7GEqPcdQPQfzTKm3dLkUNT279gbgEh8opdukSjubKrz7X): protocol: expected piece hash; context canceled; EOF
Dec 30 14:24:09.140034804 (598h55m9.01111784s) [noise=true , address=77.90.31.15:4710     , piece_id=U4QZDJXGNFNEVGYOIXCLKI3EW5EZYJWTE5KH4CVAF7DVFXXSTKTA, seg_pos={0 0}, upload=524959]: Long tail cancellation due to optimal threshold
Dec 30 14:24:09.140346318 (598h55m9.011429363s) [noise=true , address=77.90.31.11:4575     , piece_id=FCTSMJOTE6GESX2KMZIFSQQ2KVPO5KLJY4UAF2LCTCFNCPB7KAKA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc): protocol: read tcp 188.166.225.170:58170->77.90.31.11:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 30 14:24:09.140355967 (598h55m9.011439012s) [noise=true , address=77.90.31.11:4575     , piece_id=FCTSMJOTE6GESX2KMZIFSQQ2KVPO5KLJY4UAF2LCTCFNCPB7KAKA, seg_pos={0 0}, upload=524959]: Long tail cancellation due to optimal threshold
Dec 30 14:24:09.140523655 (598h55m9.0116067s) [noise=true , address=64.49.13.169:4708    , piece_id=UIEZEM3SBZAU5ZKRU6PTYXHKTGEJKUJJ7ZGKS6VI6Q76QCUPM2UA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:43936->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:43936->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43936->64.49.13.169:4708: use of closed network connection
Dec 30 14:24:09.140741952 (598h55m9.011824997s) [noise=true , address=64.49.13.169:4708    , piece_id=UIEZEM3SBZAU5ZKRU6PTYXHKTGEJKUJJ7ZGKS6VI6Q76QCUPM2UA, seg_pos={0 0}, upload=524959]: Long tail cancellation due to optimal threshold
Dec 30 14:24:09.140766057 (598h55m9.011849102s) [seg_pos={0 0}, upload=524959]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 14:24:09.140967651 (598h55m9.012050686s) [upload=524959]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 14:24:09.141080235 (598h55m9.01216328s) [upload=524959]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 14:24:09.447704404 (598h55m9.318787449s) [upload=524959]: Done uploading segments...
Dec 30 14:24:09.447727117 (598h55m9.318810162s) [upload=524959]: Upload finished. err: <nil>
Duration: 38.875796559s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
15 out of 60 is failed (threshold=3) [....X.....X...X....X......X.......X..X....X..X...X..XXX..XX.]
duration: 
5.987935533s
trace: 
https://jaeger.dev.storj.io/trace/36dad44f99247e7a
FAIL (2025-12-30 14:47:00 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ments...
Dec 30 14:45:48.177480448 (599h16m48.048563483s) [upload=525269]: Got next segment. Inline: false
Dec 30 14:45:48.177643258 (599h16m48.048726313s) [upload=525269]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 14:45:48.177650301 (599h16m48.048733336s) [upload=525269]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 14:45:48.422853313 (599h16m48.293936358s) [seg_pos={0 0}, upload=525269]: Begin upload segment...
Dec 30 14:45:48.422920802 (599h16m48.294003837s) [seg_pos={0 0}, upload=525269]: Done begin upload segment.
Dec 30 14:45:48.422950208 (599h16m48.294033243s) [noise=true , address=45.91.116.14:4189    , piece_id=7HXFSRSYQ256METQQC7B7SEPSGMRBDWUBS24GBKYP5HHR6EF7ANQ, seg_pos={0 0}, upload=525269]: Uploading piece...
Dec 30 14:45:48.423024379 (599h16m48.294107414s) [noise=true , address=145.79.227.38:4977   , piece_id=N7DN5Y236D5CGVEVNALRIKLZ3VZHMCWI4LY5OMEOICZK2RRFQJVQ, seg_pos={0 0}, upload=525269]: Uploading piece...
Dec 30 14:45:48.423775730 (599h16m48.294858775s) [noise=true , address=77.90.31.13:4629     , piece_id=ZIU5QZFTCYDJ763ZQ3IFORA7LB5QXGNM2AQTXAQBTLTIFKUE4HLQ, seg_pos={0 0}, upload=525269]: Uploading piece...
Dec 30 14:45:48.424521800 (599h16m48.295604845s) [noise=true , address=145.79.227.20:4439   , piece_id=44EYLNDP2MHNVH3V6BB4X2PAA4FLL4TP66RUQLBFGONP6CRAGYGA, seg_pos={0 0}, upload=525269]: Uploading piece...
Dec 30 14:45:48.424770584 (599h16m48.295853629s) [noise=true , address=45.91.116.6:4975     , piece_id=5FDXIGSVSWXKEHCHY2NPJVYYUVUTLQ6HSFP4RY6GQWHANQRYV2DA, seg_pos={0 0}, upload=525269]: Uploading piece...
Dec 30 14:45:48.424998518 (599h16m48.296081553s) [noise=true , address=45.91.116.10:4084    , piece_id=URR4NNW4YWY46MDEJHIMMYPFWGLEKGLC3RXMMLBXTX4COTF6FJCA, seg_pos={0 0}, upload=525269]: Uploading piece...
Dec 30 14:45:48.425444368 (599h16m48.296527413s) [noise=true , address=145.79.227.52:4446   , piece_id=5GLI3JQW3HGMSC5AHD3PDWU42TQT4SXYKP3FXZ5WH3Z4NGS7NJVQ, seg_pos={0 0}, upload=525269]: Uploading piece...
Dec 30 14:45:48.425577942 (599h16m48.296660987s) [noise=true , address=145.79.227.50:4375   , piece_id=W4QDF5DQLGG4C74U3YIHMU3PQNNNU67UOE2EYM3CO6TANR7DWFQQ, seg_pos={0 0}, upload=525269]: Uploading piece...
Dec 30 14:45:48.426069197 (599h16m48.297152232s) [noise=true , address=145.79.227.17:4970   , piece_id=HG7GSWPRJD6ALYZ77FNGJVWRUSUIDROX2VQ3EWYYDBY2X3UKBN6A, seg_pos={0 0}, upload=525269]: Uploading piece...
Dec 30 14:45:48.426162094 (599h16m48.297245139s) [noise=true , address=145.79.227.39:4014   , piece_id=WRKYHSEN6KGZMT474SPO4WFJ33YESQUJPGU6PYWFZSZC5QPGQC4Q, seg_pos={0 0}, upload=525269]: Uploading piece...
Dec 30 14:45:48.426470621 (599h16m48.297553666s) [noise=true , address=77.90.31.19:4844     , piece_id=FDSRTLAK6TTUXOVCOI4MIMCCBBPNCSUEDJAA6R4FXR7ZUGPH2RZQ, seg_pos={0 0}, upload=525269]: Uploading piece...
Dec 30 14:45:48.426717070 (599h16m48.297800106s) [noise=true , address=145.79.227.36:4518   , piece_id=SPCRCKPF5DSZZUGTJOTOZWV56WFT6MQUQUIMTIPN2DRPUIEHXDYQ, seg_pos={0 0}, upload=525269]: Uploading piece...
Dec 30 14:45:48.427027211 (599h16m48.298110256s) [noise=true , address=145.79.227.6:4257    , piece_id=BSSC6DFKQXZVQKOZNJOP4SOVCY22YTCQ6RMSIV7W2RHUSTSYAYCQ, seg_pos={0 0}, upload=525269]: Uploading piece...
Dec 30 14:45:48.427359975 (599h16m48.29844302s) [noise=true , address=77.90.31.14:4664     , piece_id=WUDQG643OQXDXXW4UYYZQJ2DQDPRIP5K4RA77CKDD4QT4TAKX4HQ, seg_pos={0 0}, upload=525269]: Uploading piece...
Dec 30 14:45:48.428091438 (599h16m48.299174473s) [noise=true , address=145.79.227.3:4149    , piece_id=4FFXO7MYQ7U5MLDAX464HVEXOUNGJI3MVOYHT5KDTD355WLVHJTA, seg_pos={0 0}, upload=525269]: Uploading piece...
Dec 30 14:45:48.428498633 (599h16m48.299581678s) [noise=true , address=145.79.227.8:4329    , piece_id=YKBRJOTUHU6NEUU7QEDRWVGB7YXL2BIZPN57KB56ILUCRR7SVXOQ, seg_pos={0 0}, upload=525269]: Uploading piece...
Dec 30 14:45:50.128610877 (599h16m49.999693922s) [upload=525269]: Next returned nil segment
Dec 30 14:45:50.128626947 (599h16m49.999709992s) [upload=525269]: Waiting for error group managing segments...
Dec 30 14:45:50.408609917 (599h16m50.279692952s) [noise=true , address=145.79.227.17:4970   , piece_id=HG7GSWPRJD6ALYZ77FNGJVWRUSUIDROX2VQ3EWYYDBY2X3UKBN6A, seg_pos={0 0}, upload=525269]: Done uploading piece. err: <nil>
Dec 30 14:45:50.421485384 (599h16m50.292568429s) [noise=true , address=145.79.227.20:4439   , piece_id=44EYLNDP2MHNVH3V6BB4X2PAA4FLL4TP66RUQLBFGONP6CRAGYGA, seg_pos={0 0}, upload=525269]: Done uploading piece. err: <nil>
Dec 30 14:45:50.421927636 (599h16m50.293010671s) [noise=true , address=145.79.227.39:4014   , piece_id=WRKYHSEN6KGZMT474SPO4WFJ33YESQUJPGU6PYWFZSZC5QPGQC4Q, seg_pos={0 0}, upload=525269]: Done uploading piece. err: <nil>
Dec 30 14:45:50.434527668 (599h16m50.305610703s) [noise=true , address=145.79.227.3:4149    , piece_id=4FFXO7MYQ7U5MLDAX464HVEXOUNGJI3MVOYHT5KDTD355WLVHJTA, seg_pos={0 0}, upload=525269]: Done uploading piece. err: <nil>
Dec 30 14:45:50.437145943 (599h16m50.308228978s) [noise=true , address=145.79.227.38:4977   , piece_id=N7DN5Y236D5CGVEVNALRIKLZ3VZHMCWI4LY5OMEOICZK2RRFQJVQ, seg_pos={0 0}, upload=525269]: Done uploading piece. err: <nil>
Dec 30 14:45:50.437361825 (599h16m50.30844487s) [noise=true , address=145.79.227.36:4518   , piece_id=SPCRCKPF5DSZZUGTJOTOZWV56WFT6MQUQUIMTIPN2DRPUIEHXDYQ, seg_pos={0 0}, upload=525269]: Done uploading piece. err: <nil>
Dec 30 14:45:50.507310808 (599h16m50.378393853s) [noise=true , address=145.79.227.52:4446   , piece_id=5GLI3JQW3HGMSC5AHD3PDWU42TQT4SXYKP3FXZ5WH3Z4NGS7NJVQ, seg_pos={0 0}, upload=525269]: Done uploading piece. err: <nil>
Dec 30 14:45:50.522382497 (599h16m50.393465542s) [noise=true , address=145.79.227.8:4329    , piece_id=YKBRJOTUHU6NEUU7QEDRWVGB7YXL2BIZPN57KB56ILUCRR7SVXOQ, seg_pos={0 0}, upload=525269]: Done uploading piece. err: <nil>
Dec 30 14:45:50.523916387 (599h16m50.394999422s) [noise=true , address=145.79.227.6:4257    , piece_id=BSSC6DFKQXZVQKOZNJOP4SOVCY22YTCQ6RMSIV7W2RHUSTSYAYCQ, seg_pos={0 0}, upload=525269]: Done uploading piece. err: <nil>
Dec 30 14:45:50.548914517 (599h16m50.419997552s) [noise=true , address=145.79.227.50:4375   , piece_id=W4QDF5DQLGG4C74U3YIHMU3PQNNNU67UOE2EYM3CO6TANR7DWFQQ, seg_pos={0 0}, upload=525269]: Done uploading piece. err: <nil>
Dec 30 14:45:50.822818631 (599h16m50.693901686s) [noise=true , address=45.91.116.6:4975     , piece_id=5FDXIGSVSWXKEHCHY2NPJVYYUVUTLQ6HSFP4RY6GQWHANQRYV2DA, seg_pos={0 0}, upload=525269]: Done uploading piece. err: <nil>
Dec 30 14:45:51.229364634 (599h16m51.100447669s) [noise=true , address=45.91.116.10:4084    , piece_id=URR4NNW4YWY46MDEJHIMMYPFWGLEKGLC3RXMMLBXTX4COTF6FJCA, seg_pos={0 0}, upload=525269]: Done uploading piece. err: <nil>
Dec 30 14:45:51.373194307 (599h16m51.244277342s) [noise=true , address=77.90.31.13:4629     , piece_id=ZIU5QZFTCYDJ763ZQ3IFORA7LB5QXGNM2AQTXAQBTLTIFKUE4HLQ, seg_pos={0 0}, upload=525269]: Done uploading piece. err: <nil>
Dec 30 14:45:51.373218744 (599h16m51.244301789s) [seg_pos={0 0}, upload=525269]: Segment reached optimal threshold of 13 pieces.
Dec 30 14:45:51.373443942 (599h16m51.244526987s) [noise=true , address=45.91.116.14:4189    , piece_id=7HXFSRSYQ256METQQC7B7SEPSGMRBDWUBS24GBKYP5HHR6EF7ANQ, seg_pos={0 0}, upload=525269]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:57656->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:57656->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57656->45.91.116.14:4189: use of closed network connection
Dec 30 14:45:51.373454382 (599h16m51.244537427s) [noise=true , address=45.91.116.14:4189    , piece_id=7HXFSRSYQ256METQQC7B7SEPSGMRBDWUBS24GBKYP5HHR6EF7ANQ, seg_pos={0 0}, upload=525269]: Long tail cancellation due to optimal threshold
Dec 30 14:45:51.373498757 (599h16m51.244581802s) [noise=true , address=77.90.31.14:4664     , piece_id=WUDQG643OQXDXXW4UYYZQJ2DQDPRIP5K4RA77CKDD4QT4TAKX4HQ, seg_pos={0 0}, upload=525269]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: write tcp 188.166.225.170:54652->77.90.31.14:4664: use of closed network connection; write tcp 188.166.225.170:54652->77.90.31.14:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54652->77.90.31.14:4664: use of closed network connection
Dec 30 14:45:51.373515739 (599h16m51.244598784s) [noise=true , address=77.90.31.14:4664     , piece_id=WUDQG643OQXDXXW4UYYZQJ2DQDPRIP5K4RA77CKDD4QT4TAKX4HQ, seg_pos={0 0}, upload=525269]: Long tail cancellation due to optimal threshold
Dec 30 14:45:51.373854494 (599h16m51.244937549s) [noise=true , address=77.90.31.19:4844     , piece_id=FDSRTLAK6TTUXOVCOI4MIMCCBBPNCSUEDJAA6R4FXR7ZUGPH2RZQ, seg_pos={0 0}, upload=525269]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF
Dec 30 14:45:51.373883971 (599h16m51.244967006s) [noise=true , address=77.90.31.19:4844     , piece_id=FDSRTLAK6TTUXOVCOI4MIMCCBBPNCSUEDJAA6R4FXR7ZUGPH2RZQ, seg_pos={0 0}, upload=525269]: Long tail cancellation due to optimal threshold
Dec 30 14:45:51.373921853 (599h16m51.245004898s) [seg_pos={0 0}, upload=525269]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 14:45:51.374024318 (599h16m51.245107363s) [upload=525269]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 14:45:51.374033335 (599h16m51.24511638s) [upload=525269]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 14:45:51.666271069 (599h16m51.537354114s) [upload=525269]: Done uploading segments...
Dec 30 14:45:51.666295586 (599h16m51.537378651s) [upload=525269]: Upload finished. err: <nil>
Duration: 5.987961361s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
46 out of 60 is failed (threshold=6) [.XXXXXXXXX.XXXXXXXXXXXXXXX.X..XX.XXXX.XXXXXXX.X..XX.X...XXXX]
duration: 
5.000651838s
trace: 
https://jaeger.dev.storj.io/trace/5d72f14e9fdcbb52
FAIL (2025-12-30 14:46:59 UTC) 
orbiter warning: context deadline exceeded
Duration: 5.000675004s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
17 out of 60 is failed (threshold=3) [XXX.XXX.....X......X......X........X.X.X..X..X.....XXX......]
duration: 
16.1428131s
trace: 
https://jaeger.dev.storj.io/trace/41ecdbaa0176ef93
FAIL (2025-12-30 14:46:53 UTC) 
orbiter warning: context deadline exceeded
Duration: 16.142836595s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)
antiblip: 
1 out of 60 is failed (threshold=3) [.................................................X..........]
duration: 
5.300063136s
trace: 
https://jaeger.dev.storj.io/trace/414189176d03adca
FAIL (2025-12-30 14:46:51 UTC) 
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 5.300094908s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)
antiblip: 
4 out of 60 is failed (threshold=3) [..X.X.X........................................X............]
duration: 
17.233697397s
trace: 
https://jaeger.dev.storj.io/trace/71938bc9cbb32ff3
FAIL (2025-12-30 14:46:47 UTC) 
orbiter warning: context deadline exceeded
Duration: 17.233722977s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
17 out of 60 is failed (threshold=3) [.XXX.XXX.....X......X......X........X.X.X..X..X.....XXX.....]
duration: 
19.501933224s
trace: 
https://jaeger.dev.storj.io/trace/41ecdbaa0176ef93
FAIL (2025-12-30 14:46:29 UTC) 
orbiter warning: context deadline exceeded
Duration: 19.501961227s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
16 out of 60 is failed (threshold=3) [X....X.....X...X....X......X.......X..X....X..X...X..XXX..XX]
duration: 
30.002765351s
trace: 
https://jaeger.dev.storj.io/trace/36dad44f99247e7a
FAIL (2025-12-30 14:46:17 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ments...
Dec 30 14:45:48.177480448 (599h16m48.048563483s) [upload=525269]: Got next segment. Inline: false
Dec 30 14:45:48.177643258 (599h16m48.048726313s) [upload=525269]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 14:45:48.177650301 (599h16m48.048733336s) [upload=525269]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 14:45:48.422853313 (599h16m48.293936358s) [seg_pos={0 0}, upload=525269]: Begin upload segment...
Dec 30 14:45:48.422920802 (599h16m48.294003837s) [seg_pos={0 0}, upload=525269]: Done begin upload segment.
Dec 30 14:45:48.422950208 (599h16m48.294033243s) [noise=true , address=45.91.116.14:4189    , piece_id=7HXFSRSYQ256METQQC7B7SEPSGMRBDWUBS24GBKYP5HHR6EF7ANQ, seg_pos={0 0}, upload=525269]: Uploading piece...
Dec 30 14:45:48.423024379 (599h16m48.294107414s) [noise=true , address=145.79.227.38:4977   , piece_id=N7DN5Y236D5CGVEVNALRIKLZ3VZHMCWI4LY5OMEOICZK2RRFQJVQ, seg_pos={0 0}, upload=525269]: Uploading piece...
Dec 30 14:45:48.423775730 (599h16m48.294858775s) [noise=true , address=77.90.31.13:4629     , piece_id=ZIU5QZFTCYDJ763ZQ3IFORA7LB5QXGNM2AQTXAQBTLTIFKUE4HLQ, seg_pos={0 0}, upload=525269]: Uploading piece...
Dec 30 14:45:48.424521800 (599h16m48.295604845s) [noise=true , address=145.79.227.20:4439   , piece_id=44EYLNDP2MHNVH3V6BB4X2PAA4FLL4TP66RUQLBFGONP6CRAGYGA, seg_pos={0 0}, upload=525269]: Uploading piece...
Dec 30 14:45:48.424770584 (599h16m48.295853629s) [noise=true , address=45.91.116.6:4975     , piece_id=5FDXIGSVSWXKEHCHY2NPJVYYUVUTLQ6HSFP4RY6GQWHANQRYV2DA, seg_pos={0 0}, upload=525269]: Uploading piece...
Dec 30 14:45:48.424998518 (599h16m48.296081553s) [noise=true , address=45.91.116.10:4084    , piece_id=URR4NNW4YWY46MDEJHIMMYPFWGLEKGLC3RXMMLBXTX4COTF6FJCA, seg_pos={0 0}, upload=525269]: Uploading piece...
Dec 30 14:45:48.425444368 (599h16m48.296527413s) [noise=true , address=145.79.227.52:4446   , piece_id=5GLI3JQW3HGMSC5AHD3PDWU42TQT4SXYKP3FXZ5WH3Z4NGS7NJVQ, seg_pos={0 0}, upload=525269]: Uploading piece...
Dec 30 14:45:48.425577942 (599h16m48.296660987s) [noise=true , address=145.79.227.50:4375   , piece_id=W4QDF5DQLGG4C74U3YIHMU3PQNNNU67UOE2EYM3CO6TANR7DWFQQ, seg_pos={0 0}, upload=525269]: Uploading piece...
Dec 30 14:45:48.426069197 (599h16m48.297152232s) [noise=true , address=145.79.227.17:4970   , piece_id=HG7GSWPRJD6ALYZ77FNGJVWRUSUIDROX2VQ3EWYYDBY2X3UKBN6A, seg_pos={0 0}, upload=525269]: Uploading piece...
Dec 30 14:45:48.426162094 (599h16m48.297245139s) [noise=true , address=145.79.227.39:4014   , piece_id=WRKYHSEN6KGZMT474SPO4WFJ33YESQUJPGU6PYWFZSZC5QPGQC4Q, seg_pos={0 0}, upload=525269]: Uploading piece...
Dec 30 14:45:48.426470621 (599h16m48.297553666s) [noise=true , address=77.90.31.19:4844     , piece_id=FDSRTLAK6TTUXOVCOI4MIMCCBBPNCSUEDJAA6R4FXR7ZUGPH2RZQ, seg_pos={0 0}, upload=525269]: Uploading piece...
Dec 30 14:45:48.426717070 (599h16m48.297800106s) [noise=true , address=145.79.227.36:4518   , piece_id=SPCRCKPF5DSZZUGTJOTOZWV56WFT6MQUQUIMTIPN2DRPUIEHXDYQ, seg_pos={0 0}, upload=525269]: Uploading piece...
Dec 30 14:45:48.427027211 (599h16m48.298110256s) [noise=true , address=145.79.227.6:4257    , piece_id=BSSC6DFKQXZVQKOZNJOP4SOVCY22YTCQ6RMSIV7W2RHUSTSYAYCQ, seg_pos={0 0}, upload=525269]: Uploading piece...
Dec 30 14:45:48.427359975 (599h16m48.29844302s) [noise=true , address=77.90.31.14:4664     , piece_id=WUDQG643OQXDXXW4UYYZQJ2DQDPRIP5K4RA77CKDD4QT4TAKX4HQ, seg_pos={0 0}, upload=525269]: Uploading piece...
Dec 30 14:45:48.428091438 (599h16m48.299174473s) [noise=true , address=145.79.227.3:4149    , piece_id=4FFXO7MYQ7U5MLDAX464HVEXOUNGJI3MVOYHT5KDTD355WLVHJTA, seg_pos={0 0}, upload=525269]: Uploading piece...
Dec 30 14:45:48.428498633 (599h16m48.299581678s) [noise=true , address=145.79.227.8:4329    , piece_id=YKBRJOTUHU6NEUU7QEDRWVGB7YXL2BIZPN57KB56ILUCRR7SVXOQ, seg_pos={0 0}, upload=525269]: Uploading piece...
Dec 30 14:45:50.128610877 (599h16m49.999693922s) [upload=525269]: Next returned nil segment
Dec 30 14:45:50.128626947 (599h16m49.999709992s) [upload=525269]: Waiting for error group managing segments...
Dec 30 14:45:50.408609917 (599h16m50.279692952s) [noise=true , address=145.79.227.17:4970   , piece_id=HG7GSWPRJD6ALYZ77FNGJVWRUSUIDROX2VQ3EWYYDBY2X3UKBN6A, seg_pos={0 0}, upload=525269]: Done uploading piece. err: <nil>
Dec 30 14:45:50.421485384 (599h16m50.292568429s) [noise=true , address=145.79.227.20:4439   , piece_id=44EYLNDP2MHNVH3V6BB4X2PAA4FLL4TP66RUQLBFGONP6CRAGYGA, seg_pos={0 0}, upload=525269]: Done uploading piece. err: <nil>
Dec 30 14:45:50.421927636 (599h16m50.293010671s) [noise=true , address=145.79.227.39:4014   , piece_id=WRKYHSEN6KGZMT474SPO4WFJ33YESQUJPGU6PYWFZSZC5QPGQC4Q, seg_pos={0 0}, upload=525269]: Done uploading piece. err: <nil>
Dec 30 14:45:50.434527668 (599h16m50.305610703s) [noise=true , address=145.79.227.3:4149    , piece_id=4FFXO7MYQ7U5MLDAX464HVEXOUNGJI3MVOYHT5KDTD355WLVHJTA, seg_pos={0 0}, upload=525269]: Done uploading piece. err: <nil>
Dec 30 14:45:50.437145943 (599h16m50.308228978s) [noise=true , address=145.79.227.38:4977   , piece_id=N7DN5Y236D5CGVEVNALRIKLZ3VZHMCWI4LY5OMEOICZK2RRFQJVQ, seg_pos={0 0}, upload=525269]: Done uploading piece. err: <nil>
Dec 30 14:45:50.437361825 (599h16m50.30844487s) [noise=true , address=145.79.227.36:4518   , piece_id=SPCRCKPF5DSZZUGTJOTOZWV56WFT6MQUQUIMTIPN2DRPUIEHXDYQ, seg_pos={0 0}, upload=525269]: Done uploading piece. err: <nil>
Dec 30 14:45:50.507310808 (599h16m50.378393853s) [noise=true , address=145.79.227.52:4446   , piece_id=5GLI3JQW3HGMSC5AHD3PDWU42TQT4SXYKP3FXZ5WH3Z4NGS7NJVQ, seg_pos={0 0}, upload=525269]: Done uploading piece. err: <nil>
Dec 30 14:45:50.522382497 (599h16m50.393465542s) [noise=true , address=145.79.227.8:4329    , piece_id=YKBRJOTUHU6NEUU7QEDRWVGB7YXL2BIZPN57KB56ILUCRR7SVXOQ, seg_pos={0 0}, upload=525269]: Done uploading piece. err: <nil>
Dec 30 14:45:50.523916387 (599h16m50.394999422s) [noise=true , address=145.79.227.6:4257    , piece_id=BSSC6DFKQXZVQKOZNJOP4SOVCY22YTCQ6RMSIV7W2RHUSTSYAYCQ, seg_pos={0 0}, upload=525269]: Done uploading piece. err: <nil>
Dec 30 14:45:50.548914517 (599h16m50.419997552s) [noise=true , address=145.79.227.50:4375   , piece_id=W4QDF5DQLGG4C74U3YIHMU3PQNNNU67UOE2EYM3CO6TANR7DWFQQ, seg_pos={0 0}, upload=525269]: Done uploading piece. err: <nil>
Dec 30 14:45:50.822818631 (599h16m50.693901686s) [noise=true , address=45.91.116.6:4975     , piece_id=5FDXIGSVSWXKEHCHY2NPJVYYUVUTLQ6HSFP4RY6GQWHANQRYV2DA, seg_pos={0 0}, upload=525269]: Done uploading piece. err: <nil>
Dec 30 14:45:51.229364634 (599h16m51.100447669s) [noise=true , address=45.91.116.10:4084    , piece_id=URR4NNW4YWY46MDEJHIMMYPFWGLEKGLC3RXMMLBXTX4COTF6FJCA, seg_pos={0 0}, upload=525269]: Done uploading piece. err: <nil>
Dec 30 14:45:51.373194307 (599h16m51.244277342s) [noise=true , address=77.90.31.13:4629     , piece_id=ZIU5QZFTCYDJ763ZQ3IFORA7LB5QXGNM2AQTXAQBTLTIFKUE4HLQ, seg_pos={0 0}, upload=525269]: Done uploading piece. err: <nil>
Dec 30 14:45:51.373218744 (599h16m51.244301789s) [seg_pos={0 0}, upload=525269]: Segment reached optimal threshold of 13 pieces.
Dec 30 14:45:51.373443942 (599h16m51.244526987s) [noise=true , address=45.91.116.14:4189    , piece_id=7HXFSRSYQ256METQQC7B7SEPSGMRBDWUBS24GBKYP5HHR6EF7ANQ, seg_pos={0 0}, upload=525269]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:57656->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:57656->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57656->45.91.116.14:4189: use of closed network connection
Dec 30 14:45:51.373454382 (599h16m51.244537427s) [noise=true , address=45.91.116.14:4189    , piece_id=7HXFSRSYQ256METQQC7B7SEPSGMRBDWUBS24GBKYP5HHR6EF7ANQ, seg_pos={0 0}, upload=525269]: Long tail cancellation due to optimal threshold
Dec 30 14:45:51.373498757 (599h16m51.244581802s) [noise=true , address=77.90.31.14:4664     , piece_id=WUDQG643OQXDXXW4UYYZQJ2DQDPRIP5K4RA77CKDD4QT4TAKX4HQ, seg_pos={0 0}, upload=525269]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: write tcp 188.166.225.170:54652->77.90.31.14:4664: use of closed network connection; write tcp 188.166.225.170:54652->77.90.31.14:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54652->77.90.31.14:4664: use of closed network connection
Dec 30 14:45:51.373515739 (599h16m51.244598784s) [noise=true , address=77.90.31.14:4664     , piece_id=WUDQG643OQXDXXW4UYYZQJ2DQDPRIP5K4RA77CKDD4QT4TAKX4HQ, seg_pos={0 0}, upload=525269]: Long tail cancellation due to optimal threshold
Dec 30 14:45:51.373854494 (599h16m51.244937549s) [noise=true , address=77.90.31.19:4844     , piece_id=FDSRTLAK6TTUXOVCOI4MIMCCBBPNCSUEDJAA6R4FXR7ZUGPH2RZQ, seg_pos={0 0}, upload=525269]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF
Dec 30 14:45:51.373883971 (599h16m51.244967006s) [noise=true , address=77.90.31.19:4844     , piece_id=FDSRTLAK6TTUXOVCOI4MIMCCBBPNCSUEDJAA6R4FXR7ZUGPH2RZQ, seg_pos={0 0}, upload=525269]: Long tail cancellation due to optimal threshold
Dec 30 14:45:51.373921853 (599h16m51.245004898s) [seg_pos={0 0}, upload=525269]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 14:45:51.374024318 (599h16m51.245107363s) [upload=525269]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 14:45:51.374033335 (599h16m51.24511638s) [upload=525269]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 14:45:51.666271069 (599h16m51.537354114s) [upload=525269]: Done uploading segments...
Dec 30 14:45:51.666295586 (599h16m51.537378651s) [upload=525269]: Upload finished. err: <nil>
Duration: 30.002786541s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
46 out of 60 is failed (threshold=6) [X.XXXXXXXXX.XXXXXXXXXXXXXXX.X..XX.XXXX.XXXXXXX.X..XX.X...XXX]
duration: 
5.001300614s
trace: 
https://jaeger.dev.storj.io/trace/32f00d26142a4070
FAIL (2025-12-30 14:46:10 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001326783s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [..................................X.........................]
duration: 
36.728733852s
trace: 
https://jaeger.dev.storj.io/trace/6a6762cfd537d983
FAIL (2025-12-30 14:46:08 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
JYHN47QHRILZESBC2RFMKJV5GL3P2MDYCEIC6OOCIMDFXHJA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.155565546 (598h55m6.026648581s) [noise=true , address=77.90.31.12:4603     , piece_id=VHIHG4OLDWDIVEUEYHK4XT744P3MW7E7KNUYNFZTQ22YX4RPARDA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.156109141 (598h55m6.027192186s) [noise=true , address=145.79.227.40:4072   , piece_id=A5BM54GXYHHDXLKFUYYR4EOCH5VGTJCAG62R4FCHGREBAFYRLKVQ, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.156505756 (598h55m6.027588801s) [noise=true , address=145.79.227.44:4208   , piece_id=PVBUF6UBX6Y3TEVO44WFXZIOXZTYZOJ4R5GPKRKUFTNOV7RUGKOQ, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.156892222 (598h55m6.027975267s) [noise=true , address=77.90.31.11:4575     , piece_id=FCTSMJOTE6GESX2KMZIFSQQ2KVPO5KLJY4UAF2LCTCFNCPB7KAKA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.157282115 (598h55m6.02836516s) [noise=true , address=64.49.13.169:4708    , piece_id=UIEZEM3SBZAU5ZKRU6PTYXHKTGEJKUJJ7ZGKS6VI6Q76QCUPM2UA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.157661989 (598h55m6.028745024s) [noise=true , address=77.90.31.21:4901     , piece_id=7QGD4I6XEQL6WRP7VFQDA53CVPB4YUANQ44II4XMCNZXD4XUUXNQ, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.158076959 (598h55m6.029160004s) [noise=true , address=77.90.31.19:4873     , piece_id=MPBWXZI4ECW54FXCK5GZLFMXJV6XVQHQFAGX6NRSUZ2IQJVVN2AQ, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.158174844 (598h55m6.029257889s) [noise=true , address=77.90.31.14:4680     , piece_id=UYYWRJXI54VNJCGACTIH3EOETSSZLMGBOEDXKY53JBQ4SYTHAQJQ, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.158483282 (598h55m6.029566327s) [noise=true , address=145.79.227.29:4958   , piece_id=HYFYCKV2LPUBYZPJ6XSBJ34OHZTCRYFERS3O6J5KADS2POL27CSA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.158804814 (598h55m6.029887859s) [noise=true , address=145.79.227.8:4343    , piece_id=CXRMAPWCX5EI4JHWJ5M6Z3QVIHZB3OBD5AZPOZN5TMKS2QDSCACQ, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.159532480 (598h55m6.030615515s) [noise=true , address=145.79.227.49:4363   , piece_id=UJBKNKMCTCEXRVTN3E77Y4DHN2LY7TFHWPOTEO7753UGFXBYYG3A, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.160163792 (598h55m6.031246837s) [noise=true , address=145.79.227.30:4768   , piece_id=GYOQQE5HMHPQCZQMLXELGSFA5ESRVFQNHHHVF25TLDH4FHCWE44A, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.160862814 (598h55m6.031945849s) [noise=true , address=145.79.227.18:4016   , piece_id=C66NGRXYHUPALVM3NM2FSHSGKX6GL7IMTUGXFIYCEJY6X4Z5PKLA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.161559340 (598h55m6.032642375s) [noise=true , address=145.79.227.34:4925   , piece_id=OAJG6P3LWWKQKNN3FXCHEY4LT4SGUC3PXYGEYY26J5TZ3XD4QBKA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.162131689 (598h55m6.033214724s) [noise=true , address=77.90.31.15:4710     , piece_id=U4QZDJXGNFNEVGYOIXCLKI3EW5EZYJWTE5KH4CVAF7DVFXXSTKTA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:07.617768952 (598h55m7.488851997s) [upload=524959]: Next returned nil segment
Dec 30 14:24:07.617782789 (598h55m7.488865824s) [upload=524959]: Waiting for error group managing segments...
Dec 30 14:24:08.019443601 (598h55m7.890526646s) [noise=true , address=145.79.227.29:4958   , piece_id=HYFYCKV2LPUBYZPJ6XSBJ34OHZTCRYFERS3O6J5KADS2POL27CSA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.054613990 (598h55m7.925697025s) [noise=true , address=145.79.227.18:4016   , piece_id=C66NGRXYHUPALVM3NM2FSHSGKX6GL7IMTUGXFIYCEJY6X4Z5PKLA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.096469250 (598h55m7.967552295s) [noise=true , address=145.79.227.8:4343    , piece_id=CXRMAPWCX5EI4JHWJ5M6Z3QVIHZB3OBD5AZPOZN5TMKS2QDSCACQ, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.342930639 (598h55m8.214013674s) [noise=true , address=77.90.31.12:4603     , piece_id=VHIHG4OLDWDIVEUEYHK4XT744P3MW7E7KNUYNFZTQ22YX4RPARDA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.344873098 (598h55m8.215956133s) [noise=true , address=77.90.31.19:4873     , piece_id=MPBWXZI4ECW54FXCK5GZLFMXJV6XVQHQFAGX6NRSUZ2IQJVVN2AQ, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.359458851 (598h55m8.230541887s) [noise=true , address=77.90.31.14:4680     , piece_id=UYYWRJXI54VNJCGACTIH3EOETSSZLMGBOEDXKY53JBQ4SYTHAQJQ, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.446410890 (598h55m8.317493925s) [noise=true , address=77.90.31.21:4901     , piece_id=7QGD4I6XEQL6WRP7VFQDA53CVPB4YUANQ44II4XMCNZXD4XUUXNQ, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.950565858 (598h55m8.821648893s) [noise=true , address=145.79.227.33:4878   , piece_id=2GGG474UM3S65BFMDV2MFWBUCYFM62Y4MWHQADZ62WHGSLBF67LA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.963965914 (598h55m8.835048949s) [noise=true , address=145.79.227.30:4768   , piece_id=GYOQQE5HMHPQCZQMLXELGSFA5ESRVFQNHHHVF25TLDH4FHCWE44A, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.987544621 (598h55m8.858627666s) [noise=true , address=145.79.227.34:4925   , piece_id=OAJG6P3LWWKQKNN3FXCHEY4LT4SGUC3PXYGEYY26J5TZ3XD4QBKA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.997751366 (598h55m8.868834411s) [noise=true , address=145.79.227.44:4208   , piece_id=PVBUF6UBX6Y3TEVO44WFXZIOXZTYZOJ4R5GPKRKUFTNOV7RUGKOQ, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:09.013743367 (598h55m8.884826412s) [noise=true , address=145.79.227.40:4072   , piece_id=A5BM54GXYHHDXLKFUYYR4EOCH5VGTJCAG62R4FCHGREBAFYRLKVQ, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:09.042854289 (598h55m8.913937334s) [noise=true , address=145.79.227.49:4363   , piece_id=UJBKNKMCTCEXRVTN3E77Y4DHN2LY7TFHWPOTEO7753UGFXBYYG3A, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:09.139174176 (598h55m9.010257221s) [noise=true , address=145.79.227.43:4174   , piece_id=NBFMEM5UTODEKLDJ7DESZVSWMYZSL3H2H4FPQE54II6SKUOMPYEA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:09.139199304 (598h55m9.010282349s) [seg_pos={0 0}, upload=524959]: Segment reached optimal threshold of 14 pieces.
Dec 30 14:24:09.139565892 (598h55m9.010648937s) [noise=true , address=77.90.31.13:4633     , piece_id=KCJXJYHN47QHRILZESBC2RFMKJV5GL3P2MDYCEIC6OOCIMDFXHJA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qXhyMbfXELW1Vg5z9NpqFX1fm3v2pGEaA56qSc8kHefPNNNYs): protocol: expected piece hash; context canceled; EOF
Dec 30 14:24:09.139575490 (598h55m9.010658535s) [noise=true , address=77.90.31.13:4633     , piece_id=KCJXJYHN47QHRILZESBC2RFMKJV5GL3P2MDYCEIC6OOCIMDFXHJA, seg_pos={0 0}, upload=524959]: Long tail cancellation due to optimal threshold
Dec 30 14:24:09.140024375 (598h55m9.01110742s) [noise=true , address=77.90.31.15:4710     , piece_id=U4QZDJXGNFNEVGYOIXCLKI3EW5EZYJWTE5KH4CVAF7DVFXXSTKTA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iHBz7GEqPcdQPQfzTKm3dLkUNT279gbgEh8opdukSjubKrz7X): protocol: expected piece hash; context canceled; EOF
Dec 30 14:24:09.140034804 (598h55m9.01111784s) [noise=true , address=77.90.31.15:4710     , piece_id=U4QZDJXGNFNEVGYOIXCLKI3EW5EZYJWTE5KH4CVAF7DVFXXSTKTA, seg_pos={0 0}, upload=524959]: Long tail cancellation due to optimal threshold
Dec 30 14:24:09.140346318 (598h55m9.011429363s) [noise=true , address=77.90.31.11:4575     , piece_id=FCTSMJOTE6GESX2KMZIFSQQ2KVPO5KLJY4UAF2LCTCFNCPB7KAKA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc): protocol: read tcp 188.166.225.170:58170->77.90.31.11:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 30 14:24:09.140355967 (598h55m9.011439012s) [noise=true , address=77.90.31.11:4575     , piece_id=FCTSMJOTE6GESX2KMZIFSQQ2KVPO5KLJY4UAF2LCTCFNCPB7KAKA, seg_pos={0 0}, upload=524959]: Long tail cancellation due to optimal threshold
Dec 30 14:24:09.140523655 (598h55m9.0116067s) [noise=true , address=64.49.13.169:4708    , piece_id=UIEZEM3SBZAU5ZKRU6PTYXHKTGEJKUJJ7ZGKS6VI6Q76QCUPM2UA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:43936->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:43936->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43936->64.49.13.169:4708: use of closed network connection
Dec 30 14:24:09.140741952 (598h55m9.011824997s) [noise=true , address=64.49.13.169:4708    , piece_id=UIEZEM3SBZAU5ZKRU6PTYXHKTGEJKUJJ7ZGKS6VI6Q76QCUPM2UA, seg_pos={0 0}, upload=524959]: Long tail cancellation due to optimal threshold
Dec 30 14:24:09.140766057 (598h55m9.011849102s) [seg_pos={0 0}, upload=524959]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 14:24:09.140967651 (598h55m9.012050686s) [upload=524959]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 14:24:09.141080235 (598h55m9.01216328s) [upload=524959]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 14:24:09.447704404 (598h55m9.318787449s) [upload=524959]: Done uploading segments...
Dec 30 14:24:09.447727117 (598h55m9.318810162s) [upload=524959]: Upload finished. err: <nil>
Duration: 36.72877976s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)
antiblip: 
1 out of 60 is failed (threshold=3) [..................................................X.........]
duration: 
13.323319412s
trace: 
https://jaeger.dev.storj.io/trace/414189176d03adca
FAIL (2025-12-30 14:45:59 UTC) 
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 13.323343668s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)
antiblip: 
4 out of 60 is failed (threshold=3) [...X.X.X........................................X...........]
duration: 
23.613363535s
trace: 
https://jaeger.dev.storj.io/trace/71938bc9cbb32ff3
FAIL (2025-12-30 14:45:50 UTC) 
orbiter warning: context deadline exceeded
Duration: 23.613386519s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
18 out of 60 is failed (threshold=3) [X.XXX.XXX.....X......X......X........X.X.X..X..X.....XXX....]
duration: 
24.942976403s
trace: 
https://jaeger.dev.storj.io/trace/41ecdbaa0176ef93
FAIL (2025-12-30 14:45:27 UTC) 
orbiter warning: context deadline exceeded
Duration: 24.943010999s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
15 out of 60 is failed (threshold=3) [.X....X.....X...X....X......X.......X..X....X..X...X..XXX..X]
duration: 
30.003490319s
trace: 
https://jaeger.dev.storj.io/trace/7f0a334558abc632
FAIL (2025-12-30 14:45:17 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: ERF3CRR4ER4K4KBMB66BNI4AOWW6A73RIDRP7XLMYDEOP2N7APLA) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: HD5MI7IQRFHQ32LZTYLWHXCJK2K3KXTYXOMQA4LOFGFG7WNLNQKQ) context deadline exceeded
======LOGS======
Dec 30 14:44:48.253245598 (599h15m48.124328633s) [upload=525255]: Starting upload
Dec 30 14:44:48.253303258 (599h15m48.124386293s) [upload=525255]: Done starting upload
Dec 30 14:44:48.253408348 (599h15m48.124491383s) [upload=525255]: Uploading segments...
Dec 30 14:44:48.253471008 (599h15m48.124554043s) [upload=525255]: Got next segment. Inline: false
Dec 30 14:44:48.253511655 (599h15m48.12459469s) [upload=525255]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 14:44:48.253516154 (599h15m48.124599189s) [upload=525255]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 14:44:48.509270147 (599h15m48.380353182s) [seg_pos={0 0}, upload=525255]: Begin upload segment...
Dec 30 14:44:48.509321956 (599h15m48.380405001s) [seg_pos={0 0}, upload=525255]: Done begin upload segment.
Dec 30 14:44:48.509362002 (599h15m48.380445047s) [noise=true , address=145.79.227.7:4294    , piece_id=6EJQCRR53TNQGWW6CF5SSSZZCNREVOZAI5DGUKYLPGWVCFENYQNQ, seg_pos={0 0}, upload=525255]: Uploading piece...
Dec 30 14:44:48.509402338 (599h15m48.380485383s) [noise=true , address=145.79.227.15:4874   , piece_id=ICWEWIXSPMYJ7XUDPJ7OLRDTXMV3BIFYENU5DMQQHUKEAGK3FJ5A, seg_pos={0 0}, upload=525255]: Uploading piece...
Dec 30 14:44:48.510013412 (599h15m48.381096447s) [noise=true , address=145.79.227.14:4837   , piece_id=UOWPFNRDBMMVOF6GKFAFIZOXO4XYVCYFWRLLFDP253GRFDPH3ENA, seg_pos={0 0}, upload=525255]: Uploading piece...
Dec 30 14:44:48.510507072 (599h15m48.381590107s) [noise=true , address=145.79.227.4:4185    , piece_id=PWTHDRDWHNH3BOLURDM4R35K57ZMF7EFNG2ODN7JOXDTGHFBNLMA, seg_pos={0 0}, upload=525255]: Uploading piece...
Dec 30 14:44:48.510946089 (599h15m48.382029134s) [noise=true , address=145.79.227.43:4157   , piece_id=FEPO26WOPRF2TLQZZCQ6WFGEICS2COMAN52RBVFC74VTEMBGDOLA, seg_pos={0 0}, upload=525255]: Uploading piece...
Dec 30 14:44:48.511266789 (599h15m48.382349824s) [noise=true , address=45.91.116.4:4937     , piece_id=SIR4NP5CJQEQDDVYGAWDFUCTACFHPQUTUL2ML7JE63QDVX33NWAQ, seg_pos={0 0}, upload=525255]: Uploading piece...
Dec 30 14:44:48.511744068 (599h15m48.382827103s) [noise=true , address=77.90.31.12:4589     , piece_id=ERF3CRR4ER4K4KBMB66BNI4AOWW6A73RIDRP7XLMYDEOP2N7APLA, seg_pos={0 0}, upload=525255]: Uploading piece...
Dec 30 14:44:48.513097605 (599h15m48.38418064s) [noise=true , address=77.90.31.10:4518     , piece_id=HD5MI7IQRFHQ32LZTYLWHXCJK2K3KXTYXOMQA4LOFGFG7WNLNQKQ, seg_pos={0 0}, upload=525255]: Uploading piece...
Dec 30 14:44:48.513471527 (599h15m48.384554562s) [noise=true , address=145.79.227.11:4729   , piece_id=H52YFEII2LQC74EY364WPODTD4SMDHR4CAOTW2KEPLCDSHDIBH7Q, seg_pos={0 0}, upload=525255]: Uploading piece...
Dec 30 14:44:48.515037930 (599h15m48.386120965s) [noise=true , address=145.79.227.32:4835   , piece_id=FHL77XCW255SR3ITYLMG3J6ZH6ZKIO7QPTDDX2RGY3FL6EXTW5TQ, seg_pos={0 0}, upload=525255]: Uploading piece...
Dec 30 14:44:48.516499142 (599h15m48.387582177s) [noise=true , address=145.79.227.17:4971   , piece_id=ASRVOVNERDKT4FGRQCQCZL3GC2GH5AIN6BQF55IVD756KYKXLEZA, seg_pos={0 0}, upload=525255]: Uploading piece...
Dec 30 14:44:48.517977508 (599h15m48.389060543s) [noise=true , address=145.79.227.16:4909   , piece_id=IT3OZV5VRWAQNHCBIW7LIJM6VKO6CLC6VKXKXMC3SMHHF5AJA2FA, seg_pos={0 0}, upload=525255]: Uploading piece...
Dec 30 14:44:48.519201789 (599h15m48.390284834s) [noise=true , address=45.91.116.10:4081    , piece_id=AQV54HEFSSU2XPE5IYAKENPZPLHECMXQRYOIFJPZ3DSSL3L3DVHA, seg_pos={0 0}, upload=525255]: Uploading piece...
Dec 30 14:44:48.519950344 (599h15m48.391033389s) [noise=true , address=45.91.116.8:4045     , piece_id=ZFZ2LSXRQC7VA74QB5RRY3W3BJ67SVCGJVRYO4PVBFAPCV2LQC5A, seg_pos={0 0}, upload=525255]: Uploading piece...
Dec 30 14:44:48.521107337 (599h15m48.392190382s) [noise=true , address=45.91.116.9:4262     , piece_id=XNPG6P367LPKJKHNIWUDN3MSUL2ZY2GWXXCFGZ3MZKIEIJVJ44HQ, seg_pos={0 0}, upload=525255]: Uploading piece...
Dec 30 14:44:48.521536905 (599h15m48.39261994s) [noise=true , address=145.79.227.9:4367    , piece_id=DIRCG46HOCROYP4NETXVIF4DRZPRIC6IITE7FBUYRH5SKA6VRYYQ, seg_pos={0 0}, upload=525255]: Uploading piece...
Dec 30 14:44:50.126974605 (599h15m49.99805764s) [upload=525255]: Next returned nil segment
Dec 30 14:44:50.126988421 (599h15m49.998071456s) [upload=525255]: Waiting for error group managing segments...
Dec 30 14:44:50.509786024 (599h15m50.380869069s) [noise=true , address=145.79.227.32:4835   , piece_id=FHL77XCW255SR3ITYLMG3J6ZH6ZKIO7QPTDDX2RGY3FL6EXTW5TQ, seg_pos={0 0}, upload=525255]: Done uploading piece. err: <nil>
Dec 30 14:44:50.514819759 (599h15m50.385902804s) [noise=true , address=145.79.227.15:4874   , piece_id=ICWEWIXSPMYJ7XUDPJ7OLRDTXMV3BIFYENU5DMQQHUKEAGK3FJ5A, seg_pos={0 0}, upload=525255]: Done uploading piece. err: <nil>
Dec 30 14:44:50.515381138 (599h15m50.386464183s) [noise=true , address=145.79.227.14:4837   , piece_id=UOWPFNRDBMMVOF6GKFAFIZOXO4XYVCYFWRLLFDP253GRFDPH3ENA, seg_pos={0 0}, upload=525255]: Done uploading piece. err: <nil>
Dec 30 14:44:50.517302456 (599h15m50.388385501s) [noise=true , address=145.79.227.7:4294    , piece_id=6EJQCRR53TNQGWW6CF5SSSZZCNREVOZAI5DGUKYLPGWVCFENYQNQ, seg_pos={0 0}, upload=525255]: Done uploading piece. err: <nil>
Dec 30 14:44:50.519727834 (599h15m50.390810879s) [noise=true , address=145.79.227.17:4971   , piece_id=ASRVOVNERDKT4FGRQCQCZL3GC2GH5AIN6BQF55IVD756KYKXLEZA, seg_pos={0 0}, upload=525255]: Done uploading piece. err: <nil>
Dec 30 14:44:50.524035687 (599h15m50.395118742s) [noise=true , address=145.79.227.4:4185    , piece_id=PWTHDRDWHNH3BOLURDM4R35K57ZMF7EFNG2ODN7JOXDTGHFBNLMA, seg_pos={0 0}, upload=525255]: Done uploading piece. err: <nil>
Dec 30 14:44:50.528718004 (599h15m50.399801079s) [noise=true , address=145.79.227.11:4729   , piece_id=H52YFEII2LQC74EY364WPODTD4SMDHR4CAOTW2KEPLCDSHDIBH7Q, seg_pos={0 0}, upload=525255]: Done uploading piece. err: <nil>
Dec 30 14:44:50.533620999 (599h15m50.404704034s) [noise=true , address=145.79.227.16:4909   , piece_id=IT3OZV5VRWAQNHCBIW7LIJM6VKO6CLC6VKXKXMC3SMHHF5AJA2FA, seg_pos={0 0}, upload=525255]: Done uploading piece. err: <nil>
Dec 30 14:44:50.534429428 (599h15m50.405512473s) [noise=true , address=145.79.227.9:4367    , piece_id=DIRCG46HOCROYP4NETXVIF4DRZPRIC6IITE7FBUYRH5SKA6VRYYQ, seg_pos={0 0}, upload=525255]: Done uploading piece. err: <nil>
Dec 30 14:44:50.703663536 (599h15m50.574746581s) [noise=true , address=77.90.31.12:4589     , piece_id=ERF3CRR4ER4K4KBMB66BNI4AOWW6A73RIDRP7XLMYDEOP2N7APLA, seg_pos={0 0}, upload=525255]: Done uploading piece. err: <nil>
Dec 30 14:44:50.798876465 (599h15m50.6699595s) [noise=true , address=77.90.31.10:4518     , piece_id=HD5MI7IQRFHQ32LZTYLWHXCJK2K3KXTYXOMQA4LOFGFG7WNLNQKQ, seg_pos={0 0}, upload=525255]: Done uploading piece. err: <nil>
Dec 30 14:44:50.852028665 (599h15m50.7231117s) [noise=true , address=45.91.116.10:4081    , piece_id=AQV54HEFSSU2XPE5IYAKENPZPLHECMXQRYOIFJPZ3DSSL3L3DVHA, seg_pos={0 0}, upload=525255]: Done uploading piece. err: <nil>
Dec 30 14:44:50.853435394 (599h15m50.724518439s) [noise=true , address=45.91.116.9:4262     , piece_id=XNPG6P367LPKJKHNIWUDN3MSUL2ZY2GWXXCFGZ3MZKIEIJVJ44HQ, seg_pos={0 0}, upload=525255]: Done uploading piece. err: <nil>
Dec 30 14:44:50.853447957 (599h15m50.724530992s) [seg_pos={0 0}, upload=525255]: Segment reached optimal threshold of 13 pieces.
Dec 30 14:44:50.853592131 (599h15m50.724675167s) [noise=true , address=45.91.116.8:4045     , piece_id=ZFZ2LSXRQC7VA74QB5RRY3W3BJ67SVCGJVRYO4PVBFAPCV2LQC5A, seg_pos={0 0}, upload=525255]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:58792->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:58792->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58792->45.91.116.8:4045: use of closed network connection
Dec 30 14:44:50.853597332 (599h15m50.724680377s) [noise=true , address=45.91.116.8:4045     , piece_id=ZFZ2LSXRQC7VA74QB5RRY3W3BJ67SVCGJVRYO4PVBFAPCV2LQC5A, seg_pos={0 0}, upload=525255]: Long tail cancellation due to optimal threshold
Dec 30 14:44:50.854710681 (599h15m50.725793726s) [noise=true , address=145.79.227.43:4157   , piece_id=FEPO26WOPRF2TLQZZCQ6WFGEICS2COMAN52RBVFC74VTEMBGDOLA, seg_pos={0 0}, upload=525255]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF
Dec 30 14:44:50.854717866 (599h15m50.725800901s) [noise=true , address=145.79.227.43:4157   , piece_id=FEPO26WOPRF2TLQZZCQ6WFGEICS2COMAN52RBVFC74VTEMBGDOLA, seg_pos={0 0}, upload=525255]: Long tail cancellation due to optimal threshold
Dec 30 14:44:50.854837042 (599h15m50.725920077s) [noise=true , address=45.91.116.4:4937     , piece_id=SIR4NP5CJQEQDDVYGAWDFUCTACFHPQUTUL2ML7JE63QDVX33NWAQ, seg_pos={0 0}, upload=525255]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF
Dec 30 14:44:50.854843805 (599h15m50.72592685s) [noise=true , address=45.91.116.4:4937     , piece_id=SIR4NP5CJQEQDDVYGAWDFUCTACFHPQUTUL2ML7JE63QDVX33NWAQ, seg_pos={0 0}, upload=525255]: Long tail cancellation due to optimal threshold
Dec 30 14:44:50.854862381 (599h15m50.725945416s) [seg_pos={0 0}, upload=525255]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 14:44:50.854932745 (599h15m50.72601584s) [upload=525255]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 14:44:50.854938716 (599h15m50.726021761s) [upload=525255]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 14:44:51.146524959 (599h15m51.017607994s) [upload=525255]: Done uploading segments...
Dec 30 14:44:51.146544076 (599h15m51.017627121s) [upload=525255]: Upload finished. err: <nil>
Duration: 30.003513404s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
46 out of 60 is failed (threshold=6) [XX.XXXXXXXXX.XXXXXXXXXXXXXXX.X..XX.XXXX.XXXXXXX.X..XX.X...XX]
duration: 
5.001511985s
trace: 
https://jaeger.dev.storj.io/trace/574e9661774bf4d6
FAIL (2025-12-30 14:45:14 UTC) 
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001538316s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)
antiblip: 
1 out of 60 is failed (threshold=3) [...................................................X........]
duration: 
6.191231659s
trace: 
https://jaeger.dev.storj.io/trace/414189176d03adca
FAIL (2025-12-30 14:44:55 UTC) 
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 6.191282106s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [...................................X........................]
duration: 
19.038081856s
trace: 
https://jaeger.dev.storj.io/trace/6a6762cfd537d983
FAIL (2025-12-30 14:44:54 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
JYHN47QHRILZESBC2RFMKJV5GL3P2MDYCEIC6OOCIMDFXHJA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.155565546 (598h55m6.026648581s) [noise=true , address=77.90.31.12:4603     , piece_id=VHIHG4OLDWDIVEUEYHK4XT744P3MW7E7KNUYNFZTQ22YX4RPARDA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.156109141 (598h55m6.027192186s) [noise=true , address=145.79.227.40:4072   , piece_id=A5BM54GXYHHDXLKFUYYR4EOCH5VGTJCAG62R4FCHGREBAFYRLKVQ, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.156505756 (598h55m6.027588801s) [noise=true , address=145.79.227.44:4208   , piece_id=PVBUF6UBX6Y3TEVO44WFXZIOXZTYZOJ4R5GPKRKUFTNOV7RUGKOQ, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.156892222 (598h55m6.027975267s) [noise=true , address=77.90.31.11:4575     , piece_id=FCTSMJOTE6GESX2KMZIFSQQ2KVPO5KLJY4UAF2LCTCFNCPB7KAKA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.157282115 (598h55m6.02836516s) [noise=true , address=64.49.13.169:4708    , piece_id=UIEZEM3SBZAU5ZKRU6PTYXHKTGEJKUJJ7ZGKS6VI6Q76QCUPM2UA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.157661989 (598h55m6.028745024s) [noise=true , address=77.90.31.21:4901     , piece_id=7QGD4I6XEQL6WRP7VFQDA53CVPB4YUANQ44II4XMCNZXD4XUUXNQ, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.158076959 (598h55m6.029160004s) [noise=true , address=77.90.31.19:4873     , piece_id=MPBWXZI4ECW54FXCK5GZLFMXJV6XVQHQFAGX6NRSUZ2IQJVVN2AQ, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.158174844 (598h55m6.029257889s) [noise=true , address=77.90.31.14:4680     , piece_id=UYYWRJXI54VNJCGACTIH3EOETSSZLMGBOEDXKY53JBQ4SYTHAQJQ, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.158483282 (598h55m6.029566327s) [noise=true , address=145.79.227.29:4958   , piece_id=HYFYCKV2LPUBYZPJ6XSBJ34OHZTCRYFERS3O6J5KADS2POL27CSA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.158804814 (598h55m6.029887859s) [noise=true , address=145.79.227.8:4343    , piece_id=CXRMAPWCX5EI4JHWJ5M6Z3QVIHZB3OBD5AZPOZN5TMKS2QDSCACQ, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.159532480 (598h55m6.030615515s) [noise=true , address=145.79.227.49:4363   , piece_id=UJBKNKMCTCEXRVTN3E77Y4DHN2LY7TFHWPOTEO7753UGFXBYYG3A, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.160163792 (598h55m6.031246837s) [noise=true , address=145.79.227.30:4768   , piece_id=GYOQQE5HMHPQCZQMLXELGSFA5ESRVFQNHHHVF25TLDH4FHCWE44A, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.160862814 (598h55m6.031945849s) [noise=true , address=145.79.227.18:4016   , piece_id=C66NGRXYHUPALVM3NM2FSHSGKX6GL7IMTUGXFIYCEJY6X4Z5PKLA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.161559340 (598h55m6.032642375s) [noise=true , address=145.79.227.34:4925   , piece_id=OAJG6P3LWWKQKNN3FXCHEY4LT4SGUC3PXYGEYY26J5TZ3XD4QBKA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.162131689 (598h55m6.033214724s) [noise=true , address=77.90.31.15:4710     , piece_id=U4QZDJXGNFNEVGYOIXCLKI3EW5EZYJWTE5KH4CVAF7DVFXXSTKTA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:07.617768952 (598h55m7.488851997s) [upload=524959]: Next returned nil segment
Dec 30 14:24:07.617782789 (598h55m7.488865824s) [upload=524959]: Waiting for error group managing segments...
Dec 30 14:24:08.019443601 (598h55m7.890526646s) [noise=true , address=145.79.227.29:4958   , piece_id=HYFYCKV2LPUBYZPJ6XSBJ34OHZTCRYFERS3O6J5KADS2POL27CSA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.054613990 (598h55m7.925697025s) [noise=true , address=145.79.227.18:4016   , piece_id=C66NGRXYHUPALVM3NM2FSHSGKX6GL7IMTUGXFIYCEJY6X4Z5PKLA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.096469250 (598h55m7.967552295s) [noise=true , address=145.79.227.8:4343    , piece_id=CXRMAPWCX5EI4JHWJ5M6Z3QVIHZB3OBD5AZPOZN5TMKS2QDSCACQ, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.342930639 (598h55m8.214013674s) [noise=true , address=77.90.31.12:4603     , piece_id=VHIHG4OLDWDIVEUEYHK4XT744P3MW7E7KNUYNFZTQ22YX4RPARDA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.344873098 (598h55m8.215956133s) [noise=true , address=77.90.31.19:4873     , piece_id=MPBWXZI4ECW54FXCK5GZLFMXJV6XVQHQFAGX6NRSUZ2IQJVVN2AQ, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.359458851 (598h55m8.230541887s) [noise=true , address=77.90.31.14:4680     , piece_id=UYYWRJXI54VNJCGACTIH3EOETSSZLMGBOEDXKY53JBQ4SYTHAQJQ, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.446410890 (598h55m8.317493925s) [noise=true , address=77.90.31.21:4901     , piece_id=7QGD4I6XEQL6WRP7VFQDA53CVPB4YUANQ44II4XMCNZXD4XUUXNQ, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.950565858 (598h55m8.821648893s) [noise=true , address=145.79.227.33:4878   , piece_id=2GGG474UM3S65BFMDV2MFWBUCYFM62Y4MWHQADZ62WHGSLBF67LA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.963965914 (598h55m8.835048949s) [noise=true , address=145.79.227.30:4768   , piece_id=GYOQQE5HMHPQCZQMLXELGSFA5ESRVFQNHHHVF25TLDH4FHCWE44A, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.987544621 (598h55m8.858627666s) [noise=true , address=145.79.227.34:4925   , piece_id=OAJG6P3LWWKQKNN3FXCHEY4LT4SGUC3PXYGEYY26J5TZ3XD4QBKA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.997751366 (598h55m8.868834411s) [noise=true , address=145.79.227.44:4208   , piece_id=PVBUF6UBX6Y3TEVO44WFXZIOXZTYZOJ4R5GPKRKUFTNOV7RUGKOQ, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:09.013743367 (598h55m8.884826412s) [noise=true , address=145.79.227.40:4072   , piece_id=A5BM54GXYHHDXLKFUYYR4EOCH5VGTJCAG62R4FCHGREBAFYRLKVQ, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:09.042854289 (598h55m8.913937334s) [noise=true , address=145.79.227.49:4363   , piece_id=UJBKNKMCTCEXRVTN3E77Y4DHN2LY7TFHWPOTEO7753UGFXBYYG3A, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:09.139174176 (598h55m9.010257221s) [noise=true , address=145.79.227.43:4174   , piece_id=NBFMEM5UTODEKLDJ7DESZVSWMYZSL3H2H4FPQE54II6SKUOMPYEA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:09.139199304 (598h55m9.010282349s) [seg_pos={0 0}, upload=524959]: Segment reached optimal threshold of 14 pieces.
Dec 30 14:24:09.139565892 (598h55m9.010648937s) [noise=true , address=77.90.31.13:4633     , piece_id=KCJXJYHN47QHRILZESBC2RFMKJV5GL3P2MDYCEIC6OOCIMDFXHJA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qXhyMbfXELW1Vg5z9NpqFX1fm3v2pGEaA56qSc8kHefPNNNYs): protocol: expected piece hash; context canceled; EOF
Dec 30 14:24:09.139575490 (598h55m9.010658535s) [noise=true , address=77.90.31.13:4633     , piece_id=KCJXJYHN47QHRILZESBC2RFMKJV5GL3P2MDYCEIC6OOCIMDFXHJA, seg_pos={0 0}, upload=524959]: Long tail cancellation due to optimal threshold
Dec 30 14:24:09.140024375 (598h55m9.01110742s) [noise=true , address=77.90.31.15:4710     , piece_id=U4QZDJXGNFNEVGYOIXCLKI3EW5EZYJWTE5KH4CVAF7DVFXXSTKTA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iHBz7GEqPcdQPQfzTKm3dLkUNT279gbgEh8opdukSjubKrz7X): protocol: expected piece hash; context canceled; EOF
Dec 30 14:24:09.140034804 (598h55m9.01111784s) [noise=true , address=77.90.31.15:4710     , piece_id=U4QZDJXGNFNEVGYOIXCLKI3EW5EZYJWTE5KH4CVAF7DVFXXSTKTA, seg_pos={0 0}, upload=524959]: Long tail cancellation due to optimal threshold
Dec 30 14:24:09.140346318 (598h55m9.011429363s) [noise=true , address=77.90.31.11:4575     , piece_id=FCTSMJOTE6GESX2KMZIFSQQ2KVPO5KLJY4UAF2LCTCFNCPB7KAKA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc): protocol: read tcp 188.166.225.170:58170->77.90.31.11:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 30 14:24:09.140355967 (598h55m9.011439012s) [noise=true , address=77.90.31.11:4575     , piece_id=FCTSMJOTE6GESX2KMZIFSQQ2KVPO5KLJY4UAF2LCTCFNCPB7KAKA, seg_pos={0 0}, upload=524959]: Long tail cancellation due to optimal threshold
Dec 30 14:24:09.140523655 (598h55m9.0116067s) [noise=true , address=64.49.13.169:4708    , piece_id=UIEZEM3SBZAU5ZKRU6PTYXHKTGEJKUJJ7ZGKS6VI6Q76QCUPM2UA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:43936->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:43936->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43936->64.49.13.169:4708: use of closed network connection
Dec 30 14:24:09.140741952 (598h55m9.011824997s) [noise=true , address=64.49.13.169:4708    , piece_id=UIEZEM3SBZAU5ZKRU6PTYXHKTGEJKUJJ7ZGKS6VI6Q76QCUPM2UA, seg_pos={0 0}, upload=524959]: Long tail cancellation due to optimal threshold
Dec 30 14:24:09.140766057 (598h55m9.011849102s) [seg_pos={0 0}, upload=524959]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 14:24:09.140967651 (598h55m9.012050686s) [upload=524959]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 14:24:09.141080235 (598h55m9.01216328s) [upload=524959]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 14:24:09.447704404 (598h55m9.318787449s) [upload=524959]: Done uploading segments...
Dec 30 14:24:09.447727117 (598h55m9.318810162s) [upload=524959]: Upload finished. err: <nil>
Duration: 19.038106973s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)
antiblip: 
5 out of 60 is failed (threshold=3) [X...X.X.X........................................X..........]
duration: 
26.556186404s
trace: 
https://jaeger.dev.storj.io/trace/71938bc9cbb32ff3
FAIL (2025-12-30 14:44:41 UTC) 
orbiter warning: context deadline exceeded
Duration: 26.556211071s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
14 out of 60 is failed (threshold=3) [..X....X.....X...X....X......X.......X..X....X..X...X..XXX..]
duration: 
5.978601367s
trace: 
https://jaeger.dev.storj.io/trace/7b4f5d70c304b5ff
FAIL (2025-12-30 14:44:30 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: EAWKMU6AZWCS6Z2673VQDI47YBFAOPMZQ2IFVE7F436WKLBFJFDQ) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: 42WJGA7ZMHSRAGQOOOO3OWYD67JOH4LBW7BOQHKXBIEXMGPVU55A) context deadline exceeded
======LOGS======
Dec 30 14:42:12.879618145 (599h13m12.75070118s) [upload=525217]: Starting upload
Dec 30 14:42:12.879695542 (599h13m12.750778598s) [upload=525217]: Done starting upload
Dec 30 14:42:12.879840359 (599h13m12.750923404s) [upload=525217]: Uploading segments...
Dec 30 14:42:12.879944507 (599h13m12.751027542s) [upload=525217]: Got next segment. Inline: false
Dec 30 14:42:12.879984042 (599h13m12.751067088s) [upload=525217]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 14:42:12.879990044 (599h13m12.751073079s) [upload=525217]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 14:42:13.169204291 (599h13m13.040287326s) [seg_pos={0 0}, upload=525217]: Begin upload segment...
Dec 30 14:42:13.169246512 (599h13m13.040329547s) [seg_pos={0 0}, upload=525217]: Done begin upload segment.
Dec 30 14:42:13.169270798 (599h13m13.040353833s) [noise=true , address=145.79.227.7:4293    , piece_id=VN6KZAZLUHB2J4KNTKADKCIEV6WMA2FIOG77QFIWGD4JFYQ35QVA, seg_pos={0 0}, upload=525217]: Uploading piece...
Dec 30 14:42:13.169404813 (599h13m13.040487849s) [noise=true , address=145.79.227.17:4971   , piece_id=J2RKOTNZ6GGBBSSGEAJCBBEOD2GM2VCEEY3NU3OR6VKYQL3CDRYA, seg_pos={0 0}, upload=525217]: Uploading piece...
Dec 30 14:42:13.170112481 (599h13m13.041195516s) [noise=true , address=145.79.227.14:4839   , piece_id=YCZYS44GLKGYMJ66NBCN2CESZ47MBWRNWD26NHRR5JNESMPFDOHQ, seg_pos={0 0}, upload=525217]: Uploading piece...
Dec 30 14:42:13.170583257 (599h13m13.041666292s) [noise=true , address=45.91.116.8:4045     , piece_id=WECWB7QVNA3B3XQQEXHCANKDLFVHAVK4XYQQCWI66WLR3BUSZZWQ, seg_pos={0 0}, upload=525217]: Uploading piece...
Dec 30 14:42:13.170947010 (599h13m13.042030045s) [noise=true , address=45.91.116.6:4974     , piece_id=VZIKLBDEJ5TVJOILOEVSYN4D22MJWCLO4EHE4QOYN3RSLJ5CPICQ, seg_pos={0 0}, upload=525217]: Uploading piece...
Dec 30 14:42:13.171320941 (599h13m13.042403976s) [noise=true , address=145.79.227.18:4006   , piece_id=YWOP4HDR73CR4XZMKGB5VEVIPVWMCNOGLFRCUAXWWWVEJ7JV4UEQ, seg_pos={0 0}, upload=525217]: Uploading piece...
Dec 30 14:42:13.171721825 (599h13m13.04280486s) [noise=true , address=145.79.227.8:4330    , piece_id=BGMY5QVXVBD2GUIJ7LT4GUPGPFFIXFASJXPH4VTHSKBGKR7BODHA, seg_pos={0 0}, upload=525217]: Uploading piece...
Dec 30 14:42:13.172083183 (599h13m13.043166228s) [noise=true , address=145.79.227.52:4445   , piece_id=FOLS7FHI6SEOI3QJ5EXYAIRUDMCG63DZRVLZISVI65JBH6EPTWUA, seg_pos={0 0}, upload=525217]: Uploading piece...
Dec 30 14:42:13.173982259 (599h13m13.045065334s) [noise=true , address=145.79.227.46:4601   , piece_id=NJBXKFR5N3QRXHQBXJS4CSUVDIOBCTQFONEMMIRPIBKGBIYXH2KQ, seg_pos={0 0}, upload=525217]: Uploading piece...
Dec 30 14:42:13.178946211 (599h13m13.050029246s) [noise=true , address=145.79.227.35:4482   , piece_id=AUJQ3RB3FXYGBHBQDTR2O4PQ6JAF2KTWMPF7WYPPNEL3VOEM23NA, seg_pos={0 0}, upload=525217]: Uploading piece...
Dec 30 14:42:13.180014676 (599h13m13.051097711s) [noise=true , address=145.79.227.25:4619   , piece_id=D66YWPHYQ7DAZXNOAIC6X7U6GDQCK657YJ5DJNGASLHUJHIJX57Q, seg_pos={0 0}, upload=525217]: Uploading piece...
Dec 30 14:42:13.182804388 (599h13m13.053887423s) [noise=true , address=145.79.227.36:4519   , piece_id=ZF4BMDRYHCDB7L7ULCG525BS6SUKWHUU3VEIIYAJXS36OWSCU6UA, seg_pos={0 0}, upload=525217]: Uploading piece...
Dec 30 14:42:13.185226560 (599h13m13.056309595s) [noise=true , address=145.79.227.20:4438   , piece_id=JGZIVXU2H36V5N4FOBNURSRKLRD7TTC2PG3SEHP24YUOO43DT42A, seg_pos={0 0}, upload=525217]: Uploading piece...
Dec 30 14:42:13.186692151 (599h13m13.057775196s) [noise=true , address=45.91.116.14:4192    , piece_id=42WJGA7ZMHSRAGQOOOO3OWYD67JOH4LBW7BOQHKXBIEXMGPVU55A, seg_pos={0 0}, upload=525217]: Uploading piece...
Dec 30 14:42:13.191043255 (599h13m13.0621263s) [noise=true , address=145.79.227.51:4410   , piece_id=OIWWVF4FVVYOEV7ZYCTPE3Y57XW7CFXU3ASGNUTEKFMH5UQQ5L3Q, seg_pos={0 0}, upload=525217]: Uploading piece...
Dec 30 14:42:13.192804479 (599h13m13.063887514s) [noise=true , address=77.90.31.17:4770     , piece_id=EAWKMU6AZWCS6Z2673VQDI47YBFAOPMZQ2IFVE7F436WKLBFJFDQ, seg_pos={0 0}, upload=525217]: Uploading piece...
Dec 30 14:42:14.915354327 (599h13m14.786437362s) [upload=525217]: Next returned nil segment
Dec 30 14:42:14.915368644 (599h13m14.786451689s) [upload=525217]: Waiting for error group managing segments...
Dec 30 14:42:15.216683342 (599h13m15.087766377s) [noise=true , address=145.79.227.7:4293    , piece_id=VN6KZAZLUHB2J4KNTKADKCIEV6WMA2FIOG77QFIWGD4JFYQ35QVA, seg_pos={0 0}, upload=525217]: Done uploading piece. err: <nil>
Dec 30 14:42:15.237697095 (599h13m15.10878014s) [noise=true , address=145.79.227.46:4601   , piece_id=NJBXKFR5N3QRXHQBXJS4CSUVDIOBCTQFONEMMIRPIBKGBIYXH2KQ, seg_pos={0 0}, upload=525217]: Done uploading piece. err: <nil>
Dec 30 14:42:15.280698786 (599h13m15.151781831s) [noise=true , address=145.79.227.20:4438   , piece_id=JGZIVXU2H36V5N4FOBNURSRKLRD7TTC2PG3SEHP24YUOO43DT42A, seg_pos={0 0}, upload=525217]: Done uploading piece. err: <nil>
Dec 30 14:42:15.331368530 (599h13m15.202451575s) [noise=true , address=145.79.227.51:4410   , piece_id=OIWWVF4FVVYOEV7ZYCTPE3Y57XW7CFXU3ASGNUTEKFMH5UQQ5L3Q, seg_pos={0 0}, upload=525217]: Done uploading piece. err: <nil>
Dec 30 14:42:15.332466069 (599h13m15.203549104s) [noise=true , address=145.79.227.18:4006   , piece_id=YWOP4HDR73CR4XZMKGB5VEVIPVWMCNOGLFRCUAXWWWVEJ7JV4UEQ, seg_pos={0 0}, upload=525217]: Done uploading piece. err: <nil>
Dec 30 14:42:15.440851130 (599h13m15.311934175s) [noise=true , address=145.79.227.8:4330    , piece_id=BGMY5QVXVBD2GUIJ7LT4GUPGPFFIXFASJXPH4VTHSKBGKR7BODHA, seg_pos={0 0}, upload=525217]: Done uploading piece. err: <nil>
Dec 30 14:42:15.471489380 (599h13m15.342572425s) [noise=true , address=145.79.227.17:4971   , piece_id=J2RKOTNZ6GGBBSSGEAJCBBEOD2GM2VCEEY3NU3OR6VKYQL3CDRYA, seg_pos={0 0}, upload=525217]: Done uploading piece. err: <nil>
Dec 30 14:42:15.479958246 (599h13m15.351041292s) [noise=true , address=145.79.227.25:4619   , piece_id=D66YWPHYQ7DAZXNOAIC6X7U6GDQCK657YJ5DJNGASLHUJHIJX57Q, seg_pos={0 0}, upload=525217]: Done uploading piece. err: <nil>
Dec 30 14:42:15.483951860 (599h13m15.355034895s) [noise=true , address=145.79.227.36:4519   , piece_id=ZF4BMDRYHCDB7L7ULCG525BS6SUKWHUU3VEIIYAJXS36OWSCU6UA, seg_pos={0 0}, upload=525217]: Done uploading piece. err: <nil>
Dec 30 14:42:15.495081232 (599h13m15.366164277s) [noise=true , address=145.79.227.52:4445   , piece_id=FOLS7FHI6SEOI3QJ5EXYAIRUDMCG63DZRVLZISVI65JBH6EPTWUA, seg_pos={0 0}, upload=525217]: Done uploading piece. err: <nil>
Dec 30 14:42:15.551929991 (599h13m15.423013036s) [noise=true , address=145.79.227.14:4839   , piece_id=YCZYS44GLKGYMJ66NBCN2CESZ47MBWRNWD26NHRR5JNESMPFDOHQ, seg_pos={0 0}, upload=525217]: Done uploading piece. err: <nil>
Dec 30 14:42:16.219057700 (599h13m16.090140745s) [noise=true , address=77.90.31.17:4770     , piece_id=EAWKMU6AZWCS6Z2673VQDI47YBFAOPMZQ2IFVE7F436WKLBFJFDQ, seg_pos={0 0}, upload=525217]: Done uploading piece. err: <nil>
Dec 30 14:42:16.351360984 (599h13m16.222444019s) [noise=true , address=45.91.116.14:4192    , piece_id=42WJGA7ZMHSRAGQOOOO3OWYD67JOH4LBW7BOQHKXBIEXMGPVU55A, seg_pos={0 0}, upload=525217]: Done uploading piece. err: <nil>
Dec 30 14:42:16.351378328 (599h13m16.222461363s) [seg_pos={0 0}, upload=525217]: Segment reached optimal threshold of 13 pieces.
Dec 30 14:42:16.351671055 (599h13m16.2227541s) [noise=true , address=145.79.227.35:4482   , piece_id=AUJQ3RB3FXYGBHBQDTR2O4PQ6JAF2KTWMPF7WYPPNEL3VOEM23NA, seg_pos={0 0}, upload=525217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF
Dec 30 14:42:16.351681314 (599h13m16.222764349s) [noise=true , address=145.79.227.35:4482   , piece_id=AUJQ3RB3FXYGBHBQDTR2O4PQ6JAF2KTWMPF7WYPPNEL3VOEM23NA, seg_pos={0 0}, upload=525217]: Long tail cancellation due to optimal threshold
Dec 30 14:42:16.351932864 (599h13m16.223015899s) [noise=true , address=45.91.116.8:4045     , piece_id=WECWB7QVNA3B3XQQEXHCANKDLFVHAVK4XYQQCWI66WLR3BUSZZWQ, seg_pos={0 0}, upload=525217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:44834->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:44834->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44834->45.91.116.8:4045: use of closed network connection
Dec 30 14:42:16.351940047 (599h13m16.223023092s) [noise=true , address=45.91.116.8:4045     , piece_id=WECWB7QVNA3B3XQQEXHCANKDLFVHAVK4XYQQCWI66WLR3BUSZZWQ, seg_pos={0 0}, upload=525217]: Long tail cancellation due to optimal threshold
Dec 30 14:42:16.352112335 (599h13m16.22319537s) [noise=true , address=45.91.116.6:4974     , piece_id=VZIKLBDEJ5TVJOILOEVSYN4D22MJWCLO4EHE4QOYN3RSLJ5CPICQ, seg_pos={0 0}, upload=525217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF
Dec 30 14:42:16.352119439 (599h13m16.223202484s) [noise=true , address=45.91.116.6:4974     , piece_id=VZIKLBDEJ5TVJOILOEVSYN4D22MJWCLO4EHE4QOYN3RSLJ5CPICQ, seg_pos={0 0}, upload=525217]: Long tail cancellation due to optimal threshold
Dec 30 14:42:16.352141922 (599h13m16.223224957s) [seg_pos={0 0}, upload=525217]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 14:42:16.352203740 (599h13m16.223286775s) [upload=525217]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 14:42:16.352209000 (599h13m16.223292035s) [upload=525217]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 14:42:16.674471915 (599h13m16.54555496s) [upload=525217]: Done uploading segments...
Dec 30 14:42:16.674495359 (599h13m16.545578394s) [upload=525217]: Upload finished. err: <nil>
Duration: 5.978627206s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
19 out of 60 is failed (threshold=3) [XX.XXX.XXX.....X......X......X........X.X.X..X..X.....XXX...]
duration: 
16.702018087s
trace: 
https://jaeger.dev.storj.io/trace/41ecdbaa0176ef93
FAIL (2025-12-30 14:44:14 UTC) 
orbiter warning: context deadline exceeded
Duration: 16.702041924s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [....................................X.......................]
duration: 
38.332798411s
trace: 
https://jaeger.dev.storj.io/trace/6a6762cfd537d983
FAIL (2025-12-30 14:44:07 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
JYHN47QHRILZESBC2RFMKJV5GL3P2MDYCEIC6OOCIMDFXHJA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.155565546 (598h55m6.026648581s) [noise=true , address=77.90.31.12:4603     , piece_id=VHIHG4OLDWDIVEUEYHK4XT744P3MW7E7KNUYNFZTQ22YX4RPARDA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.156109141 (598h55m6.027192186s) [noise=true , address=145.79.227.40:4072   , piece_id=A5BM54GXYHHDXLKFUYYR4EOCH5VGTJCAG62R4FCHGREBAFYRLKVQ, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.156505756 (598h55m6.027588801s) [noise=true , address=145.79.227.44:4208   , piece_id=PVBUF6UBX6Y3TEVO44WFXZIOXZTYZOJ4R5GPKRKUFTNOV7RUGKOQ, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.156892222 (598h55m6.027975267s) [noise=true , address=77.90.31.11:4575     , piece_id=FCTSMJOTE6GESX2KMZIFSQQ2KVPO5KLJY4UAF2LCTCFNCPB7KAKA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.157282115 (598h55m6.02836516s) [noise=true , address=64.49.13.169:4708    , piece_id=UIEZEM3SBZAU5ZKRU6PTYXHKTGEJKUJJ7ZGKS6VI6Q76QCUPM2UA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.157661989 (598h55m6.028745024s) [noise=true , address=77.90.31.21:4901     , piece_id=7QGD4I6XEQL6WRP7VFQDA53CVPB4YUANQ44II4XMCNZXD4XUUXNQ, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.158076959 (598h55m6.029160004s) [noise=true , address=77.90.31.19:4873     , piece_id=MPBWXZI4ECW54FXCK5GZLFMXJV6XVQHQFAGX6NRSUZ2IQJVVN2AQ, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.158174844 (598h55m6.029257889s) [noise=true , address=77.90.31.14:4680     , piece_id=UYYWRJXI54VNJCGACTIH3EOETSSZLMGBOEDXKY53JBQ4SYTHAQJQ, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.158483282 (598h55m6.029566327s) [noise=true , address=145.79.227.29:4958   , piece_id=HYFYCKV2LPUBYZPJ6XSBJ34OHZTCRYFERS3O6J5KADS2POL27CSA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.158804814 (598h55m6.029887859s) [noise=true , address=145.79.227.8:4343    , piece_id=CXRMAPWCX5EI4JHWJ5M6Z3QVIHZB3OBD5AZPOZN5TMKS2QDSCACQ, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.159532480 (598h55m6.030615515s) [noise=true , address=145.79.227.49:4363   , piece_id=UJBKNKMCTCEXRVTN3E77Y4DHN2LY7TFHWPOTEO7753UGFXBYYG3A, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.160163792 (598h55m6.031246837s) [noise=true , address=145.79.227.30:4768   , piece_id=GYOQQE5HMHPQCZQMLXELGSFA5ESRVFQNHHHVF25TLDH4FHCWE44A, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.160862814 (598h55m6.031945849s) [noise=true , address=145.79.227.18:4016   , piece_id=C66NGRXYHUPALVM3NM2FSHSGKX6GL7IMTUGXFIYCEJY6X4Z5PKLA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.161559340 (598h55m6.032642375s) [noise=true , address=145.79.227.34:4925   , piece_id=OAJG6P3LWWKQKNN3FXCHEY4LT4SGUC3PXYGEYY26J5TZ3XD4QBKA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:06.162131689 (598h55m6.033214724s) [noise=true , address=77.90.31.15:4710     , piece_id=U4QZDJXGNFNEVGYOIXCLKI3EW5EZYJWTE5KH4CVAF7DVFXXSTKTA, seg_pos={0 0}, upload=524959]: Uploading piece...
Dec 30 14:24:07.617768952 (598h55m7.488851997s) [upload=524959]: Next returned nil segment
Dec 30 14:24:07.617782789 (598h55m7.488865824s) [upload=524959]: Waiting for error group managing segments...
Dec 30 14:24:08.019443601 (598h55m7.890526646s) [noise=true , address=145.79.227.29:4958   , piece_id=HYFYCKV2LPUBYZPJ6XSBJ34OHZTCRYFERS3O6J5KADS2POL27CSA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.054613990 (598h55m7.925697025s) [noise=true , address=145.79.227.18:4016   , piece_id=C66NGRXYHUPALVM3NM2FSHSGKX6GL7IMTUGXFIYCEJY6X4Z5PKLA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.096469250 (598h55m7.967552295s) [noise=true , address=145.79.227.8:4343    , piece_id=CXRMAPWCX5EI4JHWJ5M6Z3QVIHZB3OBD5AZPOZN5TMKS2QDSCACQ, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.342930639 (598h55m8.214013674s) [noise=true , address=77.90.31.12:4603     , piece_id=VHIHG4OLDWDIVEUEYHK4XT744P3MW7E7KNUYNFZTQ22YX4RPARDA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.344873098 (598h55m8.215956133s) [noise=true , address=77.90.31.19:4873     , piece_id=MPBWXZI4ECW54FXCK5GZLFMXJV6XVQHQFAGX6NRSUZ2IQJVVN2AQ, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.359458851 (598h55m8.230541887s) [noise=true , address=77.90.31.14:4680     , piece_id=UYYWRJXI54VNJCGACTIH3EOETSSZLMGBOEDXKY53JBQ4SYTHAQJQ, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.446410890 (598h55m8.317493925s) [noise=true , address=77.90.31.21:4901     , piece_id=7QGD4I6XEQL6WRP7VFQDA53CVPB4YUANQ44II4XMCNZXD4XUUXNQ, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.950565858 (598h55m8.821648893s) [noise=true , address=145.79.227.33:4878   , piece_id=2GGG474UM3S65BFMDV2MFWBUCYFM62Y4MWHQADZ62WHGSLBF67LA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.963965914 (598h55m8.835048949s) [noise=true , address=145.79.227.30:4768   , piece_id=GYOQQE5HMHPQCZQMLXELGSFA5ESRVFQNHHHVF25TLDH4FHCWE44A, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.987544621 (598h55m8.858627666s) [noise=true , address=145.79.227.34:4925   , piece_id=OAJG6P3LWWKQKNN3FXCHEY4LT4SGUC3PXYGEYY26J5TZ3XD4QBKA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:08.997751366 (598h55m8.868834411s) [noise=true , address=145.79.227.44:4208   , piece_id=PVBUF6UBX6Y3TEVO44WFXZIOXZTYZOJ4R5GPKRKUFTNOV7RUGKOQ, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:09.013743367 (598h55m8.884826412s) [noise=true , address=145.79.227.40:4072   , piece_id=A5BM54GXYHHDXLKFUYYR4EOCH5VGTJCAG62R4FCHGREBAFYRLKVQ, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:09.042854289 (598h55m8.913937334s) [noise=true , address=145.79.227.49:4363   , piece_id=UJBKNKMCTCEXRVTN3E77Y4DHN2LY7TFHWPOTEO7753UGFXBYYG3A, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:09.139174176 (598h55m9.010257221s) [noise=true , address=145.79.227.43:4174   , piece_id=NBFMEM5UTODEKLDJ7DESZVSWMYZSL3H2H4FPQE54II6SKUOMPYEA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: <nil>
Dec 30 14:24:09.139199304 (598h55m9.010282349s) [seg_pos={0 0}, upload=524959]: Segment reached optimal threshold of 14 pieces.
Dec 30 14:24:09.139565892 (598h55m9.010648937s) [noise=true , address=77.90.31.13:4633     , piece_id=KCJXJYHN47QHRILZESBC2RFMKJV5GL3P2MDYCEIC6OOCIMDFXHJA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qXhyMbfXELW1Vg5z9NpqFX1fm3v2pGEaA56qSc8kHefPNNNYs): protocol: expected piece hash; context canceled; EOF
Dec 30 14:24:09.139575490 (598h55m9.010658535s) [noise=true , address=77.90.31.13:4633     , piece_id=KCJXJYHN47QHRILZESBC2RFMKJV5GL3P2MDYCEIC6OOCIMDFXHJA, seg_pos={0 0}, upload=524959]: Long tail cancellation due to optimal threshold
Dec 30 14:24:09.140024375 (598h55m9.01110742s) [noise=true , address=77.90.31.15:4710     , piece_id=U4QZDJXGNFNEVGYOIXCLKI3EW5EZYJWTE5KH4CVAF7DVFXXSTKTA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iHBz7GEqPcdQPQfzTKm3dLkUNT279gbgEh8opdukSjubKrz7X): protocol: expected piece hash; context canceled; EOF
Dec 30 14:24:09.140034804 (598h55m9.01111784s) [noise=true , address=77.90.31.15:4710     , piece_id=U4QZDJXGNFNEVGYOIXCLKI3EW5EZYJWTE5KH4CVAF7DVFXXSTKTA, seg_pos={0 0}, upload=524959]: Long tail cancellation due to optimal threshold
Dec 30 14:24:09.140346318 (598h55m9.011429363s) [noise=true , address=77.90.31.11:4575     , piece_id=FCTSMJOTE6GESX2KMZIFSQQ2KVPO5KLJY4UAF2LCTCFNCPB7KAKA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc): protocol: read tcp 188.166.225.170:58170->77.90.31.11:4575: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 30 14:24:09.140355967 (598h55m9.011439012s) [noise=true , address=77.90.31.11:4575     , piece_id=FCTSMJOTE6GESX2KMZIFSQQ2KVPO5KLJY4UAF2LCTCFNCPB7KAKA, seg_pos={0 0}, upload=524959]: Long tail cancellation due to optimal threshold
Dec 30 14:24:09.140523655 (598h55m9.0116067s) [noise=true , address=64.49.13.169:4708    , piece_id=UIEZEM3SBZAU5ZKRU6PTYXHKTGEJKUJJ7ZGKS6VI6Q76QCUPM2UA, seg_pos={0 0}, upload=524959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:43936->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:43936->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43936->64.49.13.169:4708: use of closed network connection
Dec 30 14:24:09.140741952 (598h55m9.011824997s) [noise=true , address=64.49.13.169:4708    , piece_id=UIEZEM3SBZAU5ZKRU6PTYXHKTGEJKUJJ7ZGKS6VI6Q76QCUPM2UA, seg_pos={0 0}, upload=524959]: Long tail cancellation due to optimal threshold
Dec 30 14:24:09.140766057 (598h55m9.011849102s) [seg_pos={0 0}, upload=524959]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 14:24:09.140967651 (598h55m9.012050686s) [upload=524959]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 14:24:09.141080235 (598h55m9.01216328s) [upload=524959]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 14:24:09.447704404 (598h55m9.318787449s) [upload=524959]: Done uploading segments...
Dec 30 14:24:09.447727117 (598h55m9.318810162s) [upload=524959]: Upload finished. err: <nil>
Duration: 38.332815484s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
46 out of 60 is failed (threshold=6) [XXX.XXXXXXXXX.XXXXXXXXXXXXXXX.X..XX.XXXX.XXXXXXX.X..XX.X...X]
duration: 
5.001436472s
trace: 
https://jaeger.dev.storj.io/trace/62c4fb02fb2227e6
FAIL (2025-12-30 14:44:07 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001455749s core uplink us-select-1 us1-placement-20