Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [X.X.XX....X..X.....X.X...X.X.X.X.XX..XX..X..XXX.X.......X.X.] duration: 5.77565476s trace: https://jaeger.dev.storj.io/trace/5ad0870ab909194a FAIL (2025-12-29 11:32:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: CR3FSUV7KK5U2UBYF3DJ3NC5BRWZJ7JYPYMON67SXFZW2IBQTL4Q) context deadline exceeded; piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: R5QLEDNAHAYC2CCGLEHGXD3UZDXCIGANBLWXPNBA5JM3OR2Z6T4A) context deadline exceeded; piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: X7OO5CBEOGJF5YLWDLLDSZCCS5KDY7SWXV3HDDXLTBNXRPJMLLIQ) context deadline exceeded ======LOGS====== Dec 29 11:31:13.221587680 (572h2m13.092670715s) [upload=501510]: Starting upload Dec 29 11:31:13.221652484 (572h2m13.092735539s) [upload=501510]: Done starting upload Dec 29 11:31:13.221737465 (572h2m13.0928205s) [upload=501510]: Uploading segments... Dec 29 11:31:13.221877101 (572h2m13.092960136s) [upload=501510]: Got next segment. Inline: false Dec 29 11:31:13.221914403 (572h2m13.092997448s) [upload=501510]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 11:31:13.221918521 (572h2m13.093001556s) [upload=501510]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 11:31:13.468408766 (572h2m13.339491801s) [seg_pos={0 0}, upload=501510]: Begin upload segment... Dec 29 11:31:13.468452629 (572h2m13.339535674s) [seg_pos={0 0}, upload=501510]: Done begin upload segment. Dec 29 11:31:13.468477367 (572h2m13.339560402s) [noise=true , address=77.90.31.21:4878 , piece_id=X7OO5CBEOGJF5YLWDLLDSZCCS5KDY7SWXV3HDDXLTBNXRPJMLLIQ, seg_pos={0 0}, upload=501510]: Uploading piece... Dec 29 11:31:13.468580733 (572h2m13.339663769s) [noise=true , address=145.79.227.20:4437 , piece_id=3STNVSFI4QXWWBBIDPMU3U25HUDKUTHEQNB6OF2JD5LA3TGE6M2A, seg_pos={0 0}, upload=501510]: Uploading piece... Dec 29 11:31:13.469145809 (572h2m13.340228844s) [noise=true , address=45.91.116.11:4118 , piece_id=CHVZRAWS6KN5KKTOSMMSFXOMKPQBTJSU2UGEKZAI7TDHGV4EHCJQ, seg_pos={0 0}, upload=501510]: Uploading piece... Dec 29 11:31:13.469191336 (572h2m13.340274371s) [noise=true , address=145.79.227.35:4481 , piece_id=QSSG2M5CNZUAXNB57K3GDZZY6XJSVZYZWVT6JNFS6YVZILJINEQQ, seg_pos={0 0}, upload=501510]: Uploading piece... Dec 29 11:31:13.469452453 (572h2m13.340535488s) [noise=true , address=145.79.227.12:4767 , piece_id=2SJXKSS4FFUQKPJKYUONNQTLY7JANVLZS3I4RXDNTT5V7ALV2RBQ, seg_pos={0 0}, upload=501510]: Uploading piece... Dec 29 11:31:13.469610223 (572h2m13.340693258s) [noise=true , address=145.79.227.46:4603 , piece_id=WTBVLBQLHNJ7SMB4EL5CWAS4SJW6E4BOB7WGYIRUHNBQKWB2VWTQ, seg_pos={0 0}, upload=501510]: Uploading piece... Dec 29 11:31:13.469934361 (572h2m13.341017406s) [noise=true , address=145.79.227.43:4159 , piece_id=2MAVUQQPYPI7KOGG5AZEAIDFZQ5DUU43KMBA3FCV32RZOSYFJ7DQ, seg_pos={0 0}, upload=501510]: Uploading piece... Dec 29 11:31:13.469985187 (572h2m13.341068223s) [noise=true , address=145.79.227.6:4257 , piece_id=S7HSPAWXRCV26QL5SIZI2ZF65WWPKK4LM3P62BF734424FNSITIQ, seg_pos={0 0}, upload=501510]: Uploading piece... Dec 29 11:31:13.470328020 (572h2m13.341411055s) [noise=true , address=45.91.116.14:4190 , piece_id=D3FTNZR4GSPOH7S5YICG6LSWP5LS4G25RROPA4UEV4L4D2P77QAA, seg_pos={0 0}, upload=501510]: Uploading piece... Dec 29 11:31:13.470506650 (572h2m13.341589716s) [noise=true , address=45.91.116.6:4973 , piece_id=CR3FSUV7KK5U2UBYF3DJ3NC5BRWZJ7JYPYMON67SXFZW2IBQTL4Q, seg_pos={0 0}, upload=501510]: Uploading piece... Dec 29 11:31:13.470586412 (572h2m13.341669457s) [noise=true , address=77.90.31.14:4661 , piece_id=R5QLEDNAHAYC2CCGLEHGXD3UZDXCIGANBLWXPNBA5JM3OR2Z6T4A, seg_pos={0 0}, upload=501510]: Uploading piece... Dec 29 11:31:13.470837400 (572h2m13.341920435s) [noise=true , address=145.79.227.52:4445 , piece_id=LVNENGQX4AY5MXOG276LJW3KDRAZCAQBZXRMCIVMLSKR6URAAH2Q, seg_pos={0 0}, upload=501510]: Uploading piece... Dec 29 11:31:13.470923364 (572h2m13.342006399s) [noise=true , address=145.79.227.16:4910 , piece_id=4GMR2QBEJXP6BAHSCEENZEBMBBRUW65CEYIXWVJ6XWP7NCKHXDIA, seg_pos={0 0}, upload=501510]: Uploading piece... Dec 29 11:31:13.471097455 (572h2m13.342180491s) [noise=true , address=45.91.116.5:4297 , piece_id=Y3Y2IFQZVLXEEHAD2VQ4TQZ4BMRQWIKATWFYG4P47RHBDJRNFFXA, seg_pos={0 0}, upload=501510]: Uploading piece... Dec 29 11:31:13.471296163 (572h2m13.342379208s) [noise=true , address=145.79.227.14:4839 , piece_id=CUFX6QCZ3GSEP2Z7GCN3HRSNZAKI5GGT2JW7WLS6BB4H57DCBJRA, seg_pos={0 0}, upload=501510]: Uploading piece... Dec 29 11:31:13.471664435 (572h2m13.34274747s) [noise=true , address=145.79.227.38:4977 , piece_id=5ADLX4ER72IRHYC3WBTYWNDCLLJRK6C6CUHTRPBHIPXF2C2Z5WTQ, seg_pos={0 0}, upload=501510]: Uploading piece... Dec 29 11:31:15.059183853 (572h2m14.930266908s) [upload=501510]: Next returned nil segment Dec 29 11:31:15.059233558 (572h2m14.930316593s) [upload=501510]: Waiting for error group managing segments... Dec 29 11:31:15.449791230 (572h2m15.320874265s) [noise=true , address=145.79.227.38:4977 , piece_id=5ADLX4ER72IRHYC3WBTYWNDCLLJRK6C6CUHTRPBHIPXF2C2Z5WTQ, seg_pos={0 0}, upload=501510]: Done uploading piece. err: <nil> Dec 29 11:31:15.452867838 (572h2m15.323950873s) [noise=true , address=145.79.227.6:4257 , piece_id=S7HSPAWXRCV26QL5SIZI2ZF65WWPKK4LM3P62BF734424FNSITIQ, seg_pos={0 0}, upload=501510]: Done uploading piece. err: <nil> Dec 29 11:31:15.453686467 (572h2m15.324769512s) [noise=true , address=145.79.227.14:4839 , piece_id=CUFX6QCZ3GSEP2Z7GCN3HRSNZAKI5GGT2JW7WLS6BB4H57DCBJRA, seg_pos={0 0}, upload=501510]: Done uploading piece. err: <nil> Dec 29 11:31:15.454471932 (572h2m15.325554967s) [noise=true , address=145.79.227.12:4767 , piece_id=2SJXKSS4FFUQKPJKYUONNQTLY7JANVLZS3I4RXDNTT5V7ALV2RBQ, seg_pos={0 0}, upload=501510]: Done uploading piece. err: <nil> Dec 29 11:31:15.455074861 (572h2m15.326157896s) [noise=true , address=145.79.227.52:4445 , piece_id=LVNENGQX4AY5MXOG276LJW3KDRAZCAQBZXRMCIVMLSKR6URAAH2Q, seg_pos={0 0}, upload=501510]: Done uploading piece. err: <nil> Dec 29 11:31:15.456719182 (572h2m15.327802217s) [noise=true , address=145.79.227.16:4910 , piece_id=4GMR2QBEJXP6BAHSCEENZEBMBBRUW65CEYIXWVJ6XWP7NCKHXDIA, seg_pos={0 0}, upload=501510]: Done uploading piece. err: <nil> Dec 29 11:31:15.467276384 (572h2m15.338359429s) [noise=true , address=145.79.227.46:4603 , piece_id=WTBVLBQLHNJ7SMB4EL5CWAS4SJW6E4BOB7WGYIRUHNBQKWB2VWTQ, seg_pos={0 0}, upload=501510]: Done uploading piece. err: <nil> Dec 29 11:31:15.467636740 (572h2m15.338719775s) [noise=true , address=145.79.227.20:4437 , piece_id=3STNVSFI4QXWWBBIDPMU3U25HUDKUTHEQNB6OF2JD5LA3TGE6M2A, seg_pos={0 0}, upload=501510]: Done uploading piece. err: <nil> Dec 29 11:31:15.669560384 (572h2m15.540643419s) [noise=true , address=77.90.31.14:4661 , piece_id=R5QLEDNAHAYC2CCGLEHGXD3UZDXCIGANBLWXPNBA5JM3OR2Z6T4A, seg_pos={0 0}, upload=501510]: Done uploading piece. err: <nil> Dec 29 11:31:15.719436075 (572h2m15.59051911s) [noise=true , address=45.91.116.6:4973 , piece_id=CR3FSUV7KK5U2UBYF3DJ3NC5BRWZJ7JYPYMON67SXFZW2IBQTL4Q, seg_pos={0 0}, upload=501510]: Done uploading piece. err: <nil> Dec 29 11:31:15.822906788 (572h2m15.693989823s) [noise=true , address=45.91.116.5:4297 , piece_id=Y3Y2IFQZVLXEEHAD2VQ4TQZ4BMRQWIKATWFYG4P47RHBDJRNFFXA, seg_pos={0 0}, upload=501510]: Done uploading piece. err: <nil> Dec 29 11:31:16.007879193 (572h2m15.878962238s) [noise=true , address=45.91.116.14:4190 , piece_id=D3FTNZR4GSPOH7S5YICG6LSWP5LS4G25RROPA4UEV4L4D2P77QAA, seg_pos={0 0}, upload=501510]: Done uploading piece. err: <nil> Dec 29 11:31:16.143469519 (572h2m16.014552564s) [noise=true , address=77.90.31.21:4878 , piece_id=X7OO5CBEOGJF5YLWDLLDSZCCS5KDY7SWXV3HDDXLTBNXRPJMLLIQ, seg_pos={0 0}, upload=501510]: Done uploading piece. err: <nil> Dec 29 11:31:16.143485920 (572h2m16.014568966s) [seg_pos={0 0}, upload=501510]: Segment reached optimal threshold of 13 pieces. Dec 29 11:31:16.143666965 (572h2m16.01475s) [noise=true , address=145.79.227.43:4159 , piece_id=2MAVUQQPYPI7KOGG5AZEAIDFZQ5DUU43KMBA3FCV32RZOSYFJ7DQ, seg_pos={0 0}, upload=501510]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 29 11:31:16.143672796 (572h2m16.014755841s) [noise=true , address=145.79.227.43:4159 , piece_id=2MAVUQQPYPI7KOGG5AZEAIDFZQ5DUU43KMBA3FCV32RZOSYFJ7DQ, seg_pos={0 0}, upload=501510]: Long tail cancellation due to optimal threshold Dec 29 11:31:16.143773267 (572h2m16.014856302s) [noise=true , address=45.91.116.11:4118 , piece_id=CHVZRAWS6KN5KKTOSMMSFXOMKPQBTJSU2UGEKZAI7TDHGV4EHCJQ, seg_pos={0 0}, upload=501510]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 29 11:31:16.143777636 (572h2m16.014860671s) [noise=true , address=45.91.116.11:4118 , piece_id=CHVZRAWS6KN5KKTOSMMSFXOMKPQBTJSU2UGEKZAI7TDHGV4EHCJQ, seg_pos={0 0}, upload=501510]: Long tail cancellation due to optimal threshold Dec 29 11:31:16.144083959 (572h2m16.015166994s) [noise=true , address=145.79.227.35:4481 , piece_id=QSSG2M5CNZUAXNB57K3GDZZY6XJSVZYZWVT6JNFS6YVZILJINEQQ, seg_pos={0 0}, upload=501510]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 29 11:31:16.144115068 (572h2m16.015198113s) [noise=true , address=145.79.227.35:4481 , piece_id=QSSG2M5CNZUAXNB57K3GDZZY6XJSVZYZWVT6JNFS6YVZILJINEQQ, seg_pos={0 0}, upload=501510]: Long tail cancellation due to optimal threshold Dec 29 11:31:16.144166336 (572h2m16.015249381s) [seg_pos={0 0}, upload=501510]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 11:31:16.144316733 (572h2m16.015399768s) [upload=501510]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:31:16.144325239 (572h2m16.015408274s) [upload=501510]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:31:16.440428514 (572h2m16.311511549s) [upload=501510]: Done uploading segments... Dec 29 11:31:16.440446999 (572h2m16.311530034s) [upload=501510]: Upload finished. err: <nil> Duration: 5.775695488s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [X....X...X..........X....X.......X..X...XX..XX......X.X.XX.X] duration: 5.001707379s trace: https://jaeger.dev.storj.io/trace/2aa2f454f5ae89b1 FAIL (2025-12-29 11:32:02 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001734791s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............................X...............X..X...........] duration: 37.276301351s trace: https://jaeger.dev.storj.io/trace/7e7354d8023608a8 FAIL (2025-12-29 11:31:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== ) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480097317 (571h53m39.351180352s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480405053 (571h53m39.351488098s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480712969 (571h53m39.351796014s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481019453 (571h53m39.352102488s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481029532 (571h53m39.352112567s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481559832 (571h53m39.352642877s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481731919 (571h53m39.352814954s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482078329 (571h53m39.353161364s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482099830 (571h53m39.353182865s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482434487 (571h53m39.353517532s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482480845 (571h53m39.35356388s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482841442 (571h53m39.353924477s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483163645 (571h53m39.35424668s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483532478 (571h53m39.354615513s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483914596 (571h53m39.354997631s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:40.761721106 (571h53m40.632804141s) [upload=501384]: Next returned nil segment Dec 29 11:22:40.761733349 (571h53m40.632816384s) [upload=501384]: Waiting for error group managing segments... Dec 29 11:22:41.121222862 (571h53m40.992305897s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.197311646 (571h53m41.068394691s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347247568 (571h53m41.218330613s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347917313 (571h53m41.219000348s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.348510613 (571h53m41.219593648s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.503950985 (571h53m41.37503402s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.561352025 (571h53m41.43243506s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.622447380 (571h53m41.493530425s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.645494685 (571h53m41.51657773s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.247930045 (571h53m42.11901308s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.262221890 (571h53m42.133304925s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.295598123 (571h53m42.166681158s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.307481351 (571h53m42.178564386s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325775906 (571h53m42.196858941s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325865978 (571h53m42.196949023s) [seg_pos={0 0}, upload=501384]: Segment reached optimal threshold of 14 pieces. Dec 29 11:22:42.326137074 (571h53m42.197220109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection Dec 29 11:22:42.326153074 (571h53m42.197236109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326193401 (571h53m42.197276446s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326205705 (571h53m42.19728874s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326407739 (571h53m42.197490784s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326415374 (571h53m42.197498419s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326505105 (571h53m42.19758815s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326512188 (571h53m42.197595223s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326532357 (571h53m42.197615392s) [seg_pos={0 0}, upload=501384]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 11:22:42.326671562 (571h53m42.197754597s) [upload=501384]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:22:42.326677843 (571h53m42.197760879s) [upload=501384]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:22:42.610391438 (571h53m42.481474493s) [upload=501384]: Done uploading segments... Dec 29 11:22:42.610416826 (571h53m42.481499871s) [upload=501384]: Upload finished. err: <nil> Duration: 37.276331229s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.X.X.XX....X..X.....X.X...X.X.X.X.XX..XX..X..XXX.X.......X.X] duration: 30.003679844s trace: https://jaeger.dev.storj.io/trace/5ad0870ab909194a FAIL (2025-12-29 11:31:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: CR3FSUV7KK5U2UBYF3DJ3NC5BRWZJ7JYPYMON67SXFZW2IBQTL4Q) context deadline exceeded; piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: R5QLEDNAHAYC2CCGLEHGXD3UZDXCIGANBLWXPNBA5JM3OR2Z6T4A) context deadline exceeded; piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: X7OO5CBEOGJF5YLWDLLDSZCCS5KDY7SWXV3HDDXLTBNXRPJMLLIQ) context deadline exceeded ======LOGS====== Dec 29 11:31:13.221587680 (572h2m13.092670715s) [upload=501510]: Starting upload Dec 29 11:31:13.221652484 (572h2m13.092735539s) [upload=501510]: Done starting upload Dec 29 11:31:13.221737465 (572h2m13.0928205s) [upload=501510]: Uploading segments... Dec 29 11:31:13.221877101 (572h2m13.092960136s) [upload=501510]: Got next segment. Inline: false Dec 29 11:31:13.221914403 (572h2m13.092997448s) [upload=501510]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 11:31:13.221918521 (572h2m13.093001556s) [upload=501510]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 11:31:13.468408766 (572h2m13.339491801s) [seg_pos={0 0}, upload=501510]: Begin upload segment... Dec 29 11:31:13.468452629 (572h2m13.339535674s) [seg_pos={0 0}, upload=501510]: Done begin upload segment. Dec 29 11:31:13.468477367 (572h2m13.339560402s) [noise=true , address=77.90.31.21:4878 , piece_id=X7OO5CBEOGJF5YLWDLLDSZCCS5KDY7SWXV3HDDXLTBNXRPJMLLIQ, seg_pos={0 0}, upload=501510]: Uploading piece... Dec 29 11:31:13.468580733 (572h2m13.339663769s) [noise=true , address=145.79.227.20:4437 , piece_id=3STNVSFI4QXWWBBIDPMU3U25HUDKUTHEQNB6OF2JD5LA3TGE6M2A, seg_pos={0 0}, upload=501510]: Uploading piece... Dec 29 11:31:13.469145809 (572h2m13.340228844s) [noise=true , address=45.91.116.11:4118 , piece_id=CHVZRAWS6KN5KKTOSMMSFXOMKPQBTJSU2UGEKZAI7TDHGV4EHCJQ, seg_pos={0 0}, upload=501510]: Uploading piece... Dec 29 11:31:13.469191336 (572h2m13.340274371s) [noise=true , address=145.79.227.35:4481 , piece_id=QSSG2M5CNZUAXNB57K3GDZZY6XJSVZYZWVT6JNFS6YVZILJINEQQ, seg_pos={0 0}, upload=501510]: Uploading piece... Dec 29 11:31:13.469452453 (572h2m13.340535488s) [noise=true , address=145.79.227.12:4767 , piece_id=2SJXKSS4FFUQKPJKYUONNQTLY7JANVLZS3I4RXDNTT5V7ALV2RBQ, seg_pos={0 0}, upload=501510]: Uploading piece... Dec 29 11:31:13.469610223 (572h2m13.340693258s) [noise=true , address=145.79.227.46:4603 , piece_id=WTBVLBQLHNJ7SMB4EL5CWAS4SJW6E4BOB7WGYIRUHNBQKWB2VWTQ, seg_pos={0 0}, upload=501510]: Uploading piece... Dec 29 11:31:13.469934361 (572h2m13.341017406s) [noise=true , address=145.79.227.43:4159 , piece_id=2MAVUQQPYPI7KOGG5AZEAIDFZQ5DUU43KMBA3FCV32RZOSYFJ7DQ, seg_pos={0 0}, upload=501510]: Uploading piece... Dec 29 11:31:13.469985187 (572h2m13.341068223s) [noise=true , address=145.79.227.6:4257 , piece_id=S7HSPAWXRCV26QL5SIZI2ZF65WWPKK4LM3P62BF734424FNSITIQ, seg_pos={0 0}, upload=501510]: Uploading piece... Dec 29 11:31:13.470328020 (572h2m13.341411055s) [noise=true , address=45.91.116.14:4190 , piece_id=D3FTNZR4GSPOH7S5YICG6LSWP5LS4G25RROPA4UEV4L4D2P77QAA, seg_pos={0 0}, upload=501510]: Uploading piece... Dec 29 11:31:13.470506650 (572h2m13.341589716s) [noise=true , address=45.91.116.6:4973 , piece_id=CR3FSUV7KK5U2UBYF3DJ3NC5BRWZJ7JYPYMON67SXFZW2IBQTL4Q, seg_pos={0 0}, upload=501510]: Uploading piece... Dec 29 11:31:13.470586412 (572h2m13.341669457s) [noise=true , address=77.90.31.14:4661 , piece_id=R5QLEDNAHAYC2CCGLEHGXD3UZDXCIGANBLWXPNBA5JM3OR2Z6T4A, seg_pos={0 0}, upload=501510]: Uploading piece... Dec 29 11:31:13.470837400 (572h2m13.341920435s) [noise=true , address=145.79.227.52:4445 , piece_id=LVNENGQX4AY5MXOG276LJW3KDRAZCAQBZXRMCIVMLSKR6URAAH2Q, seg_pos={0 0}, upload=501510]: Uploading piece... Dec 29 11:31:13.470923364 (572h2m13.342006399s) [noise=true , address=145.79.227.16:4910 , piece_id=4GMR2QBEJXP6BAHSCEENZEBMBBRUW65CEYIXWVJ6XWP7NCKHXDIA, seg_pos={0 0}, upload=501510]: Uploading piece... Dec 29 11:31:13.471097455 (572h2m13.342180491s) [noise=true , address=45.91.116.5:4297 , piece_id=Y3Y2IFQZVLXEEHAD2VQ4TQZ4BMRQWIKATWFYG4P47RHBDJRNFFXA, seg_pos={0 0}, upload=501510]: Uploading piece... Dec 29 11:31:13.471296163 (572h2m13.342379208s) [noise=true , address=145.79.227.14:4839 , piece_id=CUFX6QCZ3GSEP2Z7GCN3HRSNZAKI5GGT2JW7WLS6BB4H57DCBJRA, seg_pos={0 0}, upload=501510]: Uploading piece... Dec 29 11:31:13.471664435 (572h2m13.34274747s) [noise=true , address=145.79.227.38:4977 , piece_id=5ADLX4ER72IRHYC3WBTYWNDCLLJRK6C6CUHTRPBHIPXF2C2Z5WTQ, seg_pos={0 0}, upload=501510]: Uploading piece... Dec 29 11:31:15.059183853 (572h2m14.930266908s) [upload=501510]: Next returned nil segment Dec 29 11:31:15.059233558 (572h2m14.930316593s) [upload=501510]: Waiting for error group managing segments... Dec 29 11:31:15.449791230 (572h2m15.320874265s) [noise=true , address=145.79.227.38:4977 , piece_id=5ADLX4ER72IRHYC3WBTYWNDCLLJRK6C6CUHTRPBHIPXF2C2Z5WTQ, seg_pos={0 0}, upload=501510]: Done uploading piece. err: <nil> Dec 29 11:31:15.452867838 (572h2m15.323950873s) [noise=true , address=145.79.227.6:4257 , piece_id=S7HSPAWXRCV26QL5SIZI2ZF65WWPKK4LM3P62BF734424FNSITIQ, seg_pos={0 0}, upload=501510]: Done uploading piece. err: <nil> Dec 29 11:31:15.453686467 (572h2m15.324769512s) [noise=true , address=145.79.227.14:4839 , piece_id=CUFX6QCZ3GSEP2Z7GCN3HRSNZAKI5GGT2JW7WLS6BB4H57DCBJRA, seg_pos={0 0}, upload=501510]: Done uploading piece. err: <nil> Dec 29 11:31:15.454471932 (572h2m15.325554967s) [noise=true , address=145.79.227.12:4767 , piece_id=2SJXKSS4FFUQKPJKYUONNQTLY7JANVLZS3I4RXDNTT5V7ALV2RBQ, seg_pos={0 0}, upload=501510]: Done uploading piece. err: <nil> Dec 29 11:31:15.455074861 (572h2m15.326157896s) [noise=true , address=145.79.227.52:4445 , piece_id=LVNENGQX4AY5MXOG276LJW3KDRAZCAQBZXRMCIVMLSKR6URAAH2Q, seg_pos={0 0}, upload=501510]: Done uploading piece. err: <nil> Dec 29 11:31:15.456719182 (572h2m15.327802217s) [noise=true , address=145.79.227.16:4910 , piece_id=4GMR2QBEJXP6BAHSCEENZEBMBBRUW65CEYIXWVJ6XWP7NCKHXDIA, seg_pos={0 0}, upload=501510]: Done uploading piece. err: <nil> Dec 29 11:31:15.467276384 (572h2m15.338359429s) [noise=true , address=145.79.227.46:4603 , piece_id=WTBVLBQLHNJ7SMB4EL5CWAS4SJW6E4BOB7WGYIRUHNBQKWB2VWTQ, seg_pos={0 0}, upload=501510]: Done uploading piece. err: <nil> Dec 29 11:31:15.467636740 (572h2m15.338719775s) [noise=true , address=145.79.227.20:4437 , piece_id=3STNVSFI4QXWWBBIDPMU3U25HUDKUTHEQNB6OF2JD5LA3TGE6M2A, seg_pos={0 0}, upload=501510]: Done uploading piece. err: <nil> Dec 29 11:31:15.669560384 (572h2m15.540643419s) [noise=true , address=77.90.31.14:4661 , piece_id=R5QLEDNAHAYC2CCGLEHGXD3UZDXCIGANBLWXPNBA5JM3OR2Z6T4A, seg_pos={0 0}, upload=501510]: Done uploading piece. err: <nil> Dec 29 11:31:15.719436075 (572h2m15.59051911s) [noise=true , address=45.91.116.6:4973 , piece_id=CR3FSUV7KK5U2UBYF3DJ3NC5BRWZJ7JYPYMON67SXFZW2IBQTL4Q, seg_pos={0 0}, upload=501510]: Done uploading piece. err: <nil> Dec 29 11:31:15.822906788 (572h2m15.693989823s) [noise=true , address=45.91.116.5:4297 , piece_id=Y3Y2IFQZVLXEEHAD2VQ4TQZ4BMRQWIKATWFYG4P47RHBDJRNFFXA, seg_pos={0 0}, upload=501510]: Done uploading piece. err: <nil> Dec 29 11:31:16.007879193 (572h2m15.878962238s) [noise=true , address=45.91.116.14:4190 , piece_id=D3FTNZR4GSPOH7S5YICG6LSWP5LS4G25RROPA4UEV4L4D2P77QAA, seg_pos={0 0}, upload=501510]: Done uploading piece. err: <nil> Dec 29 11:31:16.143469519 (572h2m16.014552564s) [noise=true , address=77.90.31.21:4878 , piece_id=X7OO5CBEOGJF5YLWDLLDSZCCS5KDY7SWXV3HDDXLTBNXRPJMLLIQ, seg_pos={0 0}, upload=501510]: Done uploading piece. err: <nil> Dec 29 11:31:16.143485920 (572h2m16.014568966s) [seg_pos={0 0}, upload=501510]: Segment reached optimal threshold of 13 pieces. Dec 29 11:31:16.143666965 (572h2m16.01475s) [noise=true , address=145.79.227.43:4159 , piece_id=2MAVUQQPYPI7KOGG5AZEAIDFZQ5DUU43KMBA3FCV32RZOSYFJ7DQ, seg_pos={0 0}, upload=501510]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 29 11:31:16.143672796 (572h2m16.014755841s) [noise=true , address=145.79.227.43:4159 , piece_id=2MAVUQQPYPI7KOGG5AZEAIDFZQ5DUU43KMBA3FCV32RZOSYFJ7DQ, seg_pos={0 0}, upload=501510]: Long tail cancellation due to optimal threshold Dec 29 11:31:16.143773267 (572h2m16.014856302s) [noise=true , address=45.91.116.11:4118 , piece_id=CHVZRAWS6KN5KKTOSMMSFXOMKPQBTJSU2UGEKZAI7TDHGV4EHCJQ, seg_pos={0 0}, upload=501510]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 29 11:31:16.143777636 (572h2m16.014860671s) [noise=true , address=45.91.116.11:4118 , piece_id=CHVZRAWS6KN5KKTOSMMSFXOMKPQBTJSU2UGEKZAI7TDHGV4EHCJQ, seg_pos={0 0}, upload=501510]: Long tail cancellation due to optimal threshold Dec 29 11:31:16.144083959 (572h2m16.015166994s) [noise=true , address=145.79.227.35:4481 , piece_id=QSSG2M5CNZUAXNB57K3GDZZY6XJSVZYZWVT6JNFS6YVZILJINEQQ, seg_pos={0 0}, upload=501510]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 29 11:31:16.144115068 (572h2m16.015198113s) [noise=true , address=145.79.227.35:4481 , piece_id=QSSG2M5CNZUAXNB57K3GDZZY6XJSVZYZWVT6JNFS6YVZILJINEQQ, seg_pos={0 0}, upload=501510]: Long tail cancellation due to optimal threshold Dec 29 11:31:16.144166336 (572h2m16.015249381s) [seg_pos={0 0}, upload=501510]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 11:31:16.144316733 (572h2m16.015399768s) [upload=501510]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:31:16.144325239 (572h2m16.015408274s) [upload=501510]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:31:16.440428514 (572h2m16.311511549s) [upload=501510]: Done uploading segments... Dec 29 11:31:16.440446999 (572h2m16.311530034s) [upload=501510]: Upload finished. err: <nil> Duration: 30.003701825s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [.X....X...X..........X....X.......X..X...XX..XX......X.X.XX.] duration: 5.00135665s trace: https://jaeger.dev.storj.io/trace/4368939ff75f7dc6 FAIL (2025-12-29 11:31:21 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001387087s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [..X....X...X..........X....X.......X..X...XX..XX......X.X.XX] duration: 5.001031275s trace: https://jaeger.dev.storj.io/trace/4368939ff75f7dc6 FAIL (2025-12-29 11:30:42 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00105518s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............................X...............X..X..........] duration: 44.882919981s trace: https://jaeger.dev.storj.io/trace/7e7354d8023608a8 FAIL (2025-12-29 11:30:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== ) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480097317 (571h53m39.351180352s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480405053 (571h53m39.351488098s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480712969 (571h53m39.351796014s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481019453 (571h53m39.352102488s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481029532 (571h53m39.352112567s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481559832 (571h53m39.352642877s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481731919 (571h53m39.352814954s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482078329 (571h53m39.353161364s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482099830 (571h53m39.353182865s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482434487 (571h53m39.353517532s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482480845 (571h53m39.35356388s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482841442 (571h53m39.353924477s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483163645 (571h53m39.35424668s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483532478 (571h53m39.354615513s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483914596 (571h53m39.354997631s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:40.761721106 (571h53m40.632804141s) [upload=501384]: Next returned nil segment Dec 29 11:22:40.761733349 (571h53m40.632816384s) [upload=501384]: Waiting for error group managing segments... Dec 29 11:22:41.121222862 (571h53m40.992305897s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.197311646 (571h53m41.068394691s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347247568 (571h53m41.218330613s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347917313 (571h53m41.219000348s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.348510613 (571h53m41.219593648s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.503950985 (571h53m41.37503402s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.561352025 (571h53m41.43243506s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.622447380 (571h53m41.493530425s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.645494685 (571h53m41.51657773s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.247930045 (571h53m42.11901308s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.262221890 (571h53m42.133304925s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.295598123 (571h53m42.166681158s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.307481351 (571h53m42.178564386s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325775906 (571h53m42.196858941s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325865978 (571h53m42.196949023s) [seg_pos={0 0}, upload=501384]: Segment reached optimal threshold of 14 pieces. Dec 29 11:22:42.326137074 (571h53m42.197220109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection Dec 29 11:22:42.326153074 (571h53m42.197236109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326193401 (571h53m42.197276446s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326205705 (571h53m42.19728874s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326407739 (571h53m42.197490784s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326415374 (571h53m42.197498419s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326505105 (571h53m42.19758815s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326512188 (571h53m42.197595223s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326532357 (571h53m42.197615392s) [seg_pos={0 0}, upload=501384]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 11:22:42.326671562 (571h53m42.197754597s) [upload=501384]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:22:42.326677843 (571h53m42.197760879s) [upload=501384]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:22:42.610391438 (571h53m42.481474493s) [upload=501384]: Done uploading segments... Dec 29 11:22:42.610416826 (571h53m42.481499871s) [upload=501384]: Upload finished. err: <nil> Duration: 44.882936892s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [..X.X.XX....X..X.....X.X...X.X.X.X.XX..XX..X..XXX.X.......X.] duration: 5.951832784s trace: https://jaeger.dev.storj.io/trace/643973e20d272722 FAIL (2025-12-29 11:30:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 24 (572h0m25.19028307s) [upload=501486]: Uploading segments... Dec 29 11:29:25.319419233 (572h0m25.190502278s) [upload=501486]: Got next segment. Inline: false Dec 29 11:29:25.319460621 (572h0m25.190543666s) [upload=501486]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 11:29:25.319467083 (572h0m25.190550118s) [upload=501486]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 11:29:25.582961026 (572h0m25.454044071s) [seg_pos={0 0}, upload=501486]: Begin upload segment... Dec 29 11:29:25.583031650 (572h0m25.454114685s) [seg_pos={0 0}, upload=501486]: Done begin upload segment. Dec 29 11:29:25.583060165 (572h0m25.4541432s) [noise=true , address=145.79.227.45:4231 , piece_id=LJBEJCHRLWHDVI6VXDIWSTA7APXFDZXSUG5FVMMVUNANM2O2D24Q, seg_pos={0 0}, upload=501486]: Uploading piece... Dec 29 11:29:25.583117103 (572h0m25.454200138s) [noise=true , address=145.79.227.32:4835 , piece_id=WWCZIVJU2KFK6UIPAGRR3FNUSFEHGBRGKKGS5OKSH6OUIX43XMEA, seg_pos={0 0}, upload=501486]: Uploading piece... Dec 29 11:29:25.583754246 (572h0m25.454837291s) [noise=true , address=145.79.227.19:4078 , piece_id=VUXADWW6WDH6PHMH6GLTSOP7TWCWYBOQXVZGDV3M4TCP6QVWHYVQ, seg_pos={0 0}, upload=501486]: Uploading piece... Dec 29 11:29:25.584126395 (572h0m25.45520944s) [noise=true , address=45.91.116.4:4939 , piece_id=BUNY732DWKTHRRRORS5ALWRRQJLWUO4NIWO7VDD6TKF3ZYGRMHIA, seg_pos={0 0}, upload=501486]: Uploading piece... Dec 29 11:29:25.584241984 (572h0m25.455325029s) [noise=true , address=145.79.227.18:4006 , piece_id=ERDYE4O226NVA3Z36HOPCNXB6PXRQLWHOZEYKRIQ2CR3LFA4SS2A, seg_pos={0 0}, upload=501486]: Uploading piece... Dec 29 11:29:25.584581471 (572h0m25.455664516s) [noise=true , address=77.90.31.21:4877 , piece_id=OETAKVSQYL74RJ4IOS4N4NDNT2S36WRLIEMHPVEB5EVIVMGAREPQ, seg_pos={0 0}, upload=501486]: Uploading piece... Dec 29 11:29:25.584686581 (572h0m25.455769626s) [noise=true , address=145.79.227.17:4971 , piece_id=KCWMH4GYHZKXJQDTWMWXQMSZITWC67T3HTQDQNJSI5HYK6HIPL7Q, seg_pos={0 0}, upload=501486]: Uploading piece... Dec 29 11:29:25.584962046 (572h0m25.456045091s) [noise=true , address=145.79.227.52:4445 , piece_id=I2FW342VTGYWPJ6ECAQKFTJ7SIGJW4TH3WI4KSHWN7CRCUBIAB3A, seg_pos={0 0}, upload=501486]: Uploading piece... Dec 29 11:29:25.585306712 (572h0m25.456389757s) [noise=true , address=145.79.227.35:4481 , piece_id=ESNWL6KENPF4BHIUI6565UDJ5HOERHMVJIUYBGRT5PDXNBOOG6JQ, seg_pos={0 0}, upload=501486]: Uploading piece... Dec 29 11:29:25.585572849 (572h0m25.456655894s) [noise=true , address=145.79.227.39:4014 , piece_id=CY7MYXCWYFPJMWC44UXDCIGJMOAJMGQQHXL3SS7IATJ4GBIHEEXA, seg_pos={0 0}, upload=501486]: Uploading piece... Dec 29 11:29:25.585855277 (572h0m25.456938322s) [noise=true , address=45.91.116.14:4192 , piece_id=7AGELIGLRBDHMBRTBXVQ5HN5L2CKEIBE5YHLUOL5YR6SKYWRC26A, seg_pos={0 0}, upload=501486]: Uploading piece... Dec 29 11:29:25.586132575 (572h0m25.45721562s) [noise=true , address=45.91.116.7:4011 , piece_id=JTUBYCQZXDNOTJQK4XN4BRH6IN5XN5IZOWRMU3YBTUE5OE4XP2PQ, seg_pos={0 0}, upload=501486]: Uploading piece... Dec 29 11:29:25.586600476 (572h0m25.457683521s) [noise=true , address=45.91.116.12:4225 , piece_id=37JB3SPOUGNAFJV4S6FWMGL2XF4H4R66DPWHGKAMTVL3GBEQIIGQ, seg_pos={0 0}, upload=501486]: Uploading piece... Dec 29 11:29:25.586997392 (572h0m25.458080427s) [noise=true , address=145.79.227.51:4409 , piece_id=ZDKHYNCF57GO2DZV4KPWDR4R4YYDVSJHWHXNOO3B4SGXWFZRH4TA, seg_pos={0 0}, upload=501486]: Uploading piece... Dec 29 11:29:25.587438100 (572h0m25.458521146s) [noise=true , address=145.79.227.47:4266 , piece_id=WU4BKC7M34JCDRFEW6PICOE6XV452YC6XPXGYNIRPY3WOKRP45BQ, seg_pos={0 0}, upload=501486]: Uploading piece... Dec 29 11:29:25.587904359 (572h0m25.458987394s) [noise=true , address=145.79.227.37:4567 , piece_id=XWPRHN4PQWGZTMNVHWYNYTFZTH4UQJUKXRQFZ4UE5JXFVKSSXMLQ, seg_pos={0 0}, upload=501486]: Uploading piece... Dec 29 11:29:27.217522402 (572h0m27.088605437s) [upload=501486]: Next returned nil segment Dec 29 11:29:27.217536208 (572h0m27.088619243s) [upload=501486]: Waiting for error group managing segments... Dec 29 11:29:27.556291571 (572h0m27.427374606s) [noise=true , address=145.79.227.17:4971 , piece_id=KCWMH4GYHZKXJQDTWMWXQMSZITWC67T3HTQDQNJSI5HYK6HIPL7Q, seg_pos={0 0}, upload=501486]: Done uploading piece. err: <nil> Dec 29 11:29:27.558069947 (572h0m27.429152992s) [noise=true , address=145.79.227.45:4231 , piece_id=LJBEJCHRLWHDVI6VXDIWSTA7APXFDZXSUG5FVMMVUNANM2O2D24Q, seg_pos={0 0}, upload=501486]: Done uploading piece. err: <nil> Dec 29 11:29:27.562813990 (572h0m27.433897035s) [noise=true , address=145.79.227.19:4078 , piece_id=VUXADWW6WDH6PHMH6GLTSOP7TWCWYBOQXVZGDV3M4TCP6QVWHYVQ, seg_pos={0 0}, upload=501486]: Done uploading piece. err: <nil> Dec 29 11:29:27.570780530 (572h0m27.441863565s) [noise=true , address=145.79.227.51:4409 , piece_id=ZDKHYNCF57GO2DZV4KPWDR4R4YYDVSJHWHXNOO3B4SGXWFZRH4TA, seg_pos={0 0}, upload=501486]: Done uploading piece. err: <nil> Dec 29 11:29:27.571394388 (572h0m27.442477423s) [noise=true , address=145.79.227.32:4835 , piece_id=WWCZIVJU2KFK6UIPAGRR3FNUSFEHGBRGKKGS5OKSH6OUIX43XMEA, seg_pos={0 0}, upload=501486]: Done uploading piece. err: <nil> Dec 29 11:29:27.572021903 (572h0m27.443104938s) [noise=true , address=145.79.227.37:4567 , piece_id=XWPRHN4PQWGZTMNVHWYNYTFZTH4UQJUKXRQFZ4UE5JXFVKSSXMLQ, seg_pos={0 0}, upload=501486]: Done uploading piece. err: <nil> Dec 29 11:29:27.572420823 (572h0m27.443503868s) [noise=true , address=145.79.227.39:4014 , piece_id=CY7MYXCWYFPJMWC44UXDCIGJMOAJMGQQHXL3SS7IATJ4GBIHEEXA, seg_pos={0 0}, upload=501486]: Done uploading piece. err: <nil> Dec 29 11:29:27.577720915 (572h0m27.44880396s) [noise=true , address=145.79.227.18:4006 , piece_id=ERDYE4O226NVA3Z36HOPCNXB6PXRQLWHOZEYKRIQ2CR3LFA4SS2A, seg_pos={0 0}, upload=501486]: Done uploading piece. err: <nil> Dec 29 11:29:27.595090350 (572h0m27.466173385s) [noise=true , address=145.79.227.52:4445 , piece_id=I2FW342VTGYWPJ6ECAQKFTJ7SIGJW4TH3WI4KSHWN7CRCUBIAB3A, seg_pos={0 0}, upload=501486]: Done uploading piece. err: <nil> Dec 29 11:29:27.595780743 (572h0m27.466863789s) [noise=true , address=145.79.227.47:4266 , piece_id=WU4BKC7M34JCDRFEW6PICOE6XV452YC6XPXGYNIRPY3WOKRP45BQ, seg_pos={0 0}, upload=501486]: Done uploading piece. err: <nil> Dec 29 11:29:27.817319227 (572h0m27.688402272s) [noise=true , address=45.91.116.14:4192 , piece_id=7AGELIGLRBDHMBRTBXVQ5HN5L2CKEIBE5YHLUOL5YR6SKYWRC26A, seg_pos={0 0}, upload=501486]: Done uploading piece. err: <nil> Dec 29 11:29:27.825386879 (572h0m27.696469924s) [noise=true , address=45.91.116.7:4011 , piece_id=JTUBYCQZXDNOTJQK4XN4BRH6IN5XN5IZOWRMU3YBTUE5OE4XP2PQ, seg_pos={0 0}, upload=501486]: Done uploading piece. err: <nil> Dec 29 11:29:27.936585809 (572h0m27.807668844s) [noise=true , address=45.91.116.12:4225 , piece_id=37JB3SPOUGNAFJV4S6FWMGL2XF4H4R66DPWHGKAMTVL3GBEQIIGQ, seg_pos={0 0}, upload=501486]: Done uploading piece. err: <nil> Dec 29 11:29:27.936605918 (572h0m27.807688953s) [seg_pos={0 0}, upload=501486]: Segment reached optimal threshold of 13 pieces. Dec 29 11:29:27.936830665 (572h0m27.8079137s) [noise=true , address=145.79.227.35:4481 , piece_id=ESNWL6KENPF4BHIUI6565UDJ5HOERHMVJIUYBGRT5PDXNBOOG6JQ, seg_pos={0 0}, upload=501486]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 29 11:29:27.936839703 (572h0m27.807922748s) [noise=true , address=145.79.227.35:4481 , piece_id=ESNWL6KENPF4BHIUI6565UDJ5HOERHMVJIUYBGRT5PDXNBOOG6JQ, seg_pos={0 0}, upload=501486]: Long tail cancellation due to optimal threshold Dec 29 11:29:27.937012873 (572h0m27.808095918s) [noise=true , address=77.90.31.21:4877 , piece_id=OETAKVSQYL74RJ4IOS4N4NDNT2S36WRLIEMHPVEB5EVIVMGAREPQ, seg_pos={0 0}, upload=501486]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: write tcp 188.166.225.170:50568->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:50568->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50568->77.90.31.21:4877: use of closed network connection Dec 29 11:29:27.937028652 (572h0m27.808111698s) [noise=true , address=77.90.31.21:4877 , piece_id=OETAKVSQYL74RJ4IOS4N4NDNT2S36WRLIEMHPVEB5EVIVMGAREPQ, seg_pos={0 0}, upload=501486]: Long tail cancellation due to optimal threshold Dec 29 11:29:27.937098075 (572h0m27.80818111s) [noise=true , address=45.91.116.4:4939 , piece_id=BUNY732DWKTHRRRORS5ALWRRQJLWUO4NIWO7VDD6TKF3ZYGRMHIA, seg_pos={0 0}, upload=501486]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:50794->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:50794->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50794->45.91.116.4:4939: use of closed network connection Dec 29 11:29:27.937105188 (572h0m27.808188223s) [noise=true , address=45.91.116.4:4939 , piece_id=BUNY732DWKTHRRRORS5ALWRRQJLWUO4NIWO7VDD6TKF3ZYGRMHIA, seg_pos={0 0}, upload=501486]: Long tail cancellation due to optimal threshold Dec 29 11:29:27.937123363 (572h0m27.808206398s) [seg_pos={0 0}, upload=501486]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 11:29:27.937195019 (572h0m27.808278055s) [upload=501486]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:29:27.937201592 (572h0m27.808284627s) [upload=501486]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:29:28.282714652 (572h0m28.153797687s) [upload=501486]: Done uploading segments... Dec 29 11:29:28.282730902 (572h0m28.153813937s) [upload=501486]: Upload finished. err: <nil> Duration: 5.951852152s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [X..X.X.XX....X..X.....X.X...X.X.X.X.XX..XX..X..XXX.X.......X] duration: 30.003101512s trace: https://jaeger.dev.storj.io/trace/643973e20d272722 FAIL (2025-12-29 11:29:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 24 (572h0m25.19028307s) [upload=501486]: Uploading segments... Dec 29 11:29:25.319419233 (572h0m25.190502278s) [upload=501486]: Got next segment. Inline: false Dec 29 11:29:25.319460621 (572h0m25.190543666s) [upload=501486]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 11:29:25.319467083 (572h0m25.190550118s) [upload=501486]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 11:29:25.582961026 (572h0m25.454044071s) [seg_pos={0 0}, upload=501486]: Begin upload segment... Dec 29 11:29:25.583031650 (572h0m25.454114685s) [seg_pos={0 0}, upload=501486]: Done begin upload segment. Dec 29 11:29:25.583060165 (572h0m25.4541432s) [noise=true , address=145.79.227.45:4231 , piece_id=LJBEJCHRLWHDVI6VXDIWSTA7APXFDZXSUG5FVMMVUNANM2O2D24Q, seg_pos={0 0}, upload=501486]: Uploading piece... Dec 29 11:29:25.583117103 (572h0m25.454200138s) [noise=true , address=145.79.227.32:4835 , piece_id=WWCZIVJU2KFK6UIPAGRR3FNUSFEHGBRGKKGS5OKSH6OUIX43XMEA, seg_pos={0 0}, upload=501486]: Uploading piece... Dec 29 11:29:25.583754246 (572h0m25.454837291s) [noise=true , address=145.79.227.19:4078 , piece_id=VUXADWW6WDH6PHMH6GLTSOP7TWCWYBOQXVZGDV3M4TCP6QVWHYVQ, seg_pos={0 0}, upload=501486]: Uploading piece... Dec 29 11:29:25.584126395 (572h0m25.45520944s) [noise=true , address=45.91.116.4:4939 , piece_id=BUNY732DWKTHRRRORS5ALWRRQJLWUO4NIWO7VDD6TKF3ZYGRMHIA, seg_pos={0 0}, upload=501486]: Uploading piece... Dec 29 11:29:25.584241984 (572h0m25.455325029s) [noise=true , address=145.79.227.18:4006 , piece_id=ERDYE4O226NVA3Z36HOPCNXB6PXRQLWHOZEYKRIQ2CR3LFA4SS2A, seg_pos={0 0}, upload=501486]: Uploading piece... Dec 29 11:29:25.584581471 (572h0m25.455664516s) [noise=true , address=77.90.31.21:4877 , piece_id=OETAKVSQYL74RJ4IOS4N4NDNT2S36WRLIEMHPVEB5EVIVMGAREPQ, seg_pos={0 0}, upload=501486]: Uploading piece... Dec 29 11:29:25.584686581 (572h0m25.455769626s) [noise=true , address=145.79.227.17:4971 , piece_id=KCWMH4GYHZKXJQDTWMWXQMSZITWC67T3HTQDQNJSI5HYK6HIPL7Q, seg_pos={0 0}, upload=501486]: Uploading piece... Dec 29 11:29:25.584962046 (572h0m25.456045091s) [noise=true , address=145.79.227.52:4445 , piece_id=I2FW342VTGYWPJ6ECAQKFTJ7SIGJW4TH3WI4KSHWN7CRCUBIAB3A, seg_pos={0 0}, upload=501486]: Uploading piece... Dec 29 11:29:25.585306712 (572h0m25.456389757s) [noise=true , address=145.79.227.35:4481 , piece_id=ESNWL6KENPF4BHIUI6565UDJ5HOERHMVJIUYBGRT5PDXNBOOG6JQ, seg_pos={0 0}, upload=501486]: Uploading piece... Dec 29 11:29:25.585572849 (572h0m25.456655894s) [noise=true , address=145.79.227.39:4014 , piece_id=CY7MYXCWYFPJMWC44UXDCIGJMOAJMGQQHXL3SS7IATJ4GBIHEEXA, seg_pos={0 0}, upload=501486]: Uploading piece... Dec 29 11:29:25.585855277 (572h0m25.456938322s) [noise=true , address=45.91.116.14:4192 , piece_id=7AGELIGLRBDHMBRTBXVQ5HN5L2CKEIBE5YHLUOL5YR6SKYWRC26A, seg_pos={0 0}, upload=501486]: Uploading piece... Dec 29 11:29:25.586132575 (572h0m25.45721562s) [noise=true , address=45.91.116.7:4011 , piece_id=JTUBYCQZXDNOTJQK4XN4BRH6IN5XN5IZOWRMU3YBTUE5OE4XP2PQ, seg_pos={0 0}, upload=501486]: Uploading piece... Dec 29 11:29:25.586600476 (572h0m25.457683521s) [noise=true , address=45.91.116.12:4225 , piece_id=37JB3SPOUGNAFJV4S6FWMGL2XF4H4R66DPWHGKAMTVL3GBEQIIGQ, seg_pos={0 0}, upload=501486]: Uploading piece... Dec 29 11:29:25.586997392 (572h0m25.458080427s) [noise=true , address=145.79.227.51:4409 , piece_id=ZDKHYNCF57GO2DZV4KPWDR4R4YYDVSJHWHXNOO3B4SGXWFZRH4TA, seg_pos={0 0}, upload=501486]: Uploading piece... Dec 29 11:29:25.587438100 (572h0m25.458521146s) [noise=true , address=145.79.227.47:4266 , piece_id=WU4BKC7M34JCDRFEW6PICOE6XV452YC6XPXGYNIRPY3WOKRP45BQ, seg_pos={0 0}, upload=501486]: Uploading piece... Dec 29 11:29:25.587904359 (572h0m25.458987394s) [noise=true , address=145.79.227.37:4567 , piece_id=XWPRHN4PQWGZTMNVHWYNYTFZTH4UQJUKXRQFZ4UE5JXFVKSSXMLQ, seg_pos={0 0}, upload=501486]: Uploading piece... Dec 29 11:29:27.217522402 (572h0m27.088605437s) [upload=501486]: Next returned nil segment Dec 29 11:29:27.217536208 (572h0m27.088619243s) [upload=501486]: Waiting for error group managing segments... Dec 29 11:29:27.556291571 (572h0m27.427374606s) [noise=true , address=145.79.227.17:4971 , piece_id=KCWMH4GYHZKXJQDTWMWXQMSZITWC67T3HTQDQNJSI5HYK6HIPL7Q, seg_pos={0 0}, upload=501486]: Done uploading piece. err: <nil> Dec 29 11:29:27.558069947 (572h0m27.429152992s) [noise=true , address=145.79.227.45:4231 , piece_id=LJBEJCHRLWHDVI6VXDIWSTA7APXFDZXSUG5FVMMVUNANM2O2D24Q, seg_pos={0 0}, upload=501486]: Done uploading piece. err: <nil> Dec 29 11:29:27.562813990 (572h0m27.433897035s) [noise=true , address=145.79.227.19:4078 , piece_id=VUXADWW6WDH6PHMH6GLTSOP7TWCWYBOQXVZGDV3M4TCP6QVWHYVQ, seg_pos={0 0}, upload=501486]: Done uploading piece. err: <nil> Dec 29 11:29:27.570780530 (572h0m27.441863565s) [noise=true , address=145.79.227.51:4409 , piece_id=ZDKHYNCF57GO2DZV4KPWDR4R4YYDVSJHWHXNOO3B4SGXWFZRH4TA, seg_pos={0 0}, upload=501486]: Done uploading piece. err: <nil> Dec 29 11:29:27.571394388 (572h0m27.442477423s) [noise=true , address=145.79.227.32:4835 , piece_id=WWCZIVJU2KFK6UIPAGRR3FNUSFEHGBRGKKGS5OKSH6OUIX43XMEA, seg_pos={0 0}, upload=501486]: Done uploading piece. err: <nil> Dec 29 11:29:27.572021903 (572h0m27.443104938s) [noise=true , address=145.79.227.37:4567 , piece_id=XWPRHN4PQWGZTMNVHWYNYTFZTH4UQJUKXRQFZ4UE5JXFVKSSXMLQ, seg_pos={0 0}, upload=501486]: Done uploading piece. err: <nil> Dec 29 11:29:27.572420823 (572h0m27.443503868s) [noise=true , address=145.79.227.39:4014 , piece_id=CY7MYXCWYFPJMWC44UXDCIGJMOAJMGQQHXL3SS7IATJ4GBIHEEXA, seg_pos={0 0}, upload=501486]: Done uploading piece. err: <nil> Dec 29 11:29:27.577720915 (572h0m27.44880396s) [noise=true , address=145.79.227.18:4006 , piece_id=ERDYE4O226NVA3Z36HOPCNXB6PXRQLWHOZEYKRIQ2CR3LFA4SS2A, seg_pos={0 0}, upload=501486]: Done uploading piece. err: <nil> Dec 29 11:29:27.595090350 (572h0m27.466173385s) [noise=true , address=145.79.227.52:4445 , piece_id=I2FW342VTGYWPJ6ECAQKFTJ7SIGJW4TH3WI4KSHWN7CRCUBIAB3A, seg_pos={0 0}, upload=501486]: Done uploading piece. err: <nil> Dec 29 11:29:27.595780743 (572h0m27.466863789s) [noise=true , address=145.79.227.47:4266 , piece_id=WU4BKC7M34JCDRFEW6PICOE6XV452YC6XPXGYNIRPY3WOKRP45BQ, seg_pos={0 0}, upload=501486]: Done uploading piece. err: <nil> Dec 29 11:29:27.817319227 (572h0m27.688402272s) [noise=true , address=45.91.116.14:4192 , piece_id=7AGELIGLRBDHMBRTBXVQ5HN5L2CKEIBE5YHLUOL5YR6SKYWRC26A, seg_pos={0 0}, upload=501486]: Done uploading piece. err: <nil> Dec 29 11:29:27.825386879 (572h0m27.696469924s) [noise=true , address=45.91.116.7:4011 , piece_id=JTUBYCQZXDNOTJQK4XN4BRH6IN5XN5IZOWRMU3YBTUE5OE4XP2PQ, seg_pos={0 0}, upload=501486]: Done uploading piece. err: <nil> Dec 29 11:29:27.936585809 (572h0m27.807668844s) [noise=true , address=45.91.116.12:4225 , piece_id=37JB3SPOUGNAFJV4S6FWMGL2XF4H4R66DPWHGKAMTVL3GBEQIIGQ, seg_pos={0 0}, upload=501486]: Done uploading piece. err: <nil> Dec 29 11:29:27.936605918 (572h0m27.807688953s) [seg_pos={0 0}, upload=501486]: Segment reached optimal threshold of 13 pieces. Dec 29 11:29:27.936830665 (572h0m27.8079137s) [noise=true , address=145.79.227.35:4481 , piece_id=ESNWL6KENPF4BHIUI6565UDJ5HOERHMVJIUYBGRT5PDXNBOOG6JQ, seg_pos={0 0}, upload=501486]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 29 11:29:27.936839703 (572h0m27.807922748s) [noise=true , address=145.79.227.35:4481 , piece_id=ESNWL6KENPF4BHIUI6565UDJ5HOERHMVJIUYBGRT5PDXNBOOG6JQ, seg_pos={0 0}, upload=501486]: Long tail cancellation due to optimal threshold Dec 29 11:29:27.937012873 (572h0m27.808095918s) [noise=true , address=77.90.31.21:4877 , piece_id=OETAKVSQYL74RJ4IOS4N4NDNT2S36WRLIEMHPVEB5EVIVMGAREPQ, seg_pos={0 0}, upload=501486]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: write tcp 188.166.225.170:50568->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:50568->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50568->77.90.31.21:4877: use of closed network connection Dec 29 11:29:27.937028652 (572h0m27.808111698s) [noise=true , address=77.90.31.21:4877 , piece_id=OETAKVSQYL74RJ4IOS4N4NDNT2S36WRLIEMHPVEB5EVIVMGAREPQ, seg_pos={0 0}, upload=501486]: Long tail cancellation due to optimal threshold Dec 29 11:29:27.937098075 (572h0m27.80818111s) [noise=true , address=45.91.116.4:4939 , piece_id=BUNY732DWKTHRRRORS5ALWRRQJLWUO4NIWO7VDD6TKF3ZYGRMHIA, seg_pos={0 0}, upload=501486]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:50794->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:50794->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50794->45.91.116.4:4939: use of closed network connection Dec 29 11:29:27.937105188 (572h0m27.808188223s) [noise=true , address=45.91.116.4:4939 , piece_id=BUNY732DWKTHRRRORS5ALWRRQJLWUO4NIWO7VDD6TKF3ZYGRMHIA, seg_pos={0 0}, upload=501486]: Long tail cancellation due to optimal threshold Dec 29 11:29:27.937123363 (572h0m27.808206398s) [seg_pos={0 0}, upload=501486]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 11:29:27.937195019 (572h0m27.808278055s) [upload=501486]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:29:27.937201592 (572h0m27.808284627s) [upload=501486]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:29:28.282714652 (572h0m28.153797687s) [upload=501486]: Done uploading segments... Dec 29 11:29:28.282730902 (572h0m28.153813937s) [upload=501486]: Upload finished. err: <nil> Duration: 30.003123584s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............................X...............X..X.........] duration: 42.198221212s trace: https://jaeger.dev.storj.io/trace/7e7354d8023608a8 FAIL (2025-12-29 11:29:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== ) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480097317 (571h53m39.351180352s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480405053 (571h53m39.351488098s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480712969 (571h53m39.351796014s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481019453 (571h53m39.352102488s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481029532 (571h53m39.352112567s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481559832 (571h53m39.352642877s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481731919 (571h53m39.352814954s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482078329 (571h53m39.353161364s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482099830 (571h53m39.353182865s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482434487 (571h53m39.353517532s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482480845 (571h53m39.35356388s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482841442 (571h53m39.353924477s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483163645 (571h53m39.35424668s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483532478 (571h53m39.354615513s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483914596 (571h53m39.354997631s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:40.761721106 (571h53m40.632804141s) [upload=501384]: Next returned nil segment Dec 29 11:22:40.761733349 (571h53m40.632816384s) [upload=501384]: Waiting for error group managing segments... Dec 29 11:22:41.121222862 (571h53m40.992305897s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.197311646 (571h53m41.068394691s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347247568 (571h53m41.218330613s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347917313 (571h53m41.219000348s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.348510613 (571h53m41.219593648s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.503950985 (571h53m41.37503402s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.561352025 (571h53m41.43243506s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.622447380 (571h53m41.493530425s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.645494685 (571h53m41.51657773s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.247930045 (571h53m42.11901308s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.262221890 (571h53m42.133304925s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.295598123 (571h53m42.166681158s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.307481351 (571h53m42.178564386s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325775906 (571h53m42.196858941s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325865978 (571h53m42.196949023s) [seg_pos={0 0}, upload=501384]: Segment reached optimal threshold of 14 pieces. Dec 29 11:22:42.326137074 (571h53m42.197220109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection Dec 29 11:22:42.326153074 (571h53m42.197236109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326193401 (571h53m42.197276446s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326205705 (571h53m42.19728874s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326407739 (571h53m42.197490784s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326415374 (571h53m42.197498419s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326505105 (571h53m42.19758815s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326512188 (571h53m42.197595223s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326532357 (571h53m42.197615392s) [seg_pos={0 0}, upload=501384]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 11:22:42.326671562 (571h53m42.197754597s) [upload=501384]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:22:42.326677843 (571h53m42.197760879s) [upload=501384]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:22:42.610391438 (571h53m42.481474493s) [upload=501384]: Done uploading segments... Dec 29 11:22:42.610416826 (571h53m42.481499871s) [upload=501384]: Upload finished. err: <nil> Duration: 42.198250989s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [...X....X...X..........X....X.......X..X...XX..XX......X.X.X] duration: 5.000700853s trace: https://jaeger.dev.storj.io/trace/69fe8edb553a76c1 FAIL (2025-12-29 11:29:32 UTC) orbiter warning: context deadline exceeded Duration: 5.00072097s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [................................X...............X..X........] duration: 27.529632297s trace: https://jaeger.dev.storj.io/trace/7e7354d8023608a8 FAIL (2025-12-29 11:28:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== ) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480097317 (571h53m39.351180352s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480405053 (571h53m39.351488098s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480712969 (571h53m39.351796014s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481019453 (571h53m39.352102488s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481029532 (571h53m39.352112567s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481559832 (571h53m39.352642877s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481731919 (571h53m39.352814954s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482078329 (571h53m39.353161364s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482099830 (571h53m39.353182865s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482434487 (571h53m39.353517532s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482480845 (571h53m39.35356388s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482841442 (571h53m39.353924477s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483163645 (571h53m39.35424668s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483532478 (571h53m39.354615513s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483914596 (571h53m39.354997631s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:40.761721106 (571h53m40.632804141s) [upload=501384]: Next returned nil segment Dec 29 11:22:40.761733349 (571h53m40.632816384s) [upload=501384]: Waiting for error group managing segments... Dec 29 11:22:41.121222862 (571h53m40.992305897s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.197311646 (571h53m41.068394691s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347247568 (571h53m41.218330613s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347917313 (571h53m41.219000348s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.348510613 (571h53m41.219593648s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.503950985 (571h53m41.37503402s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.561352025 (571h53m41.43243506s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.622447380 (571h53m41.493530425s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.645494685 (571h53m41.51657773s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.247930045 (571h53m42.11901308s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.262221890 (571h53m42.133304925s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.295598123 (571h53m42.166681158s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.307481351 (571h53m42.178564386s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325775906 (571h53m42.196858941s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325865978 (571h53m42.196949023s) [seg_pos={0 0}, upload=501384]: Segment reached optimal threshold of 14 pieces. Dec 29 11:22:42.326137074 (571h53m42.197220109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection Dec 29 11:22:42.326153074 (571h53m42.197236109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326193401 (571h53m42.197276446s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326205705 (571h53m42.19728874s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326407739 (571h53m42.197490784s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326415374 (571h53m42.197498419s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326505105 (571h53m42.19758815s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326512188 (571h53m42.197595223s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326532357 (571h53m42.197615392s) [seg_pos={0 0}, upload=501384]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 11:22:42.326671562 (571h53m42.197754597s) [upload=501384]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:22:42.326677843 (571h53m42.197760879s) [upload=501384]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:22:42.610391438 (571h53m42.481474493s) [upload=501384]: Done uploading segments... Dec 29 11:22:42.610416826 (571h53m42.481499871s) [upload=501384]: Upload finished. err: <nil> Duration: 27.529648186s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [X...X....X...X..........X....X.......X..X...XX..XX......X.X.] duration: 3.449385871s trace: https://jaeger.dev.storj.io/trace/76f9311d5d4a38d0 FAIL (2025-12-29 11:28:32 UTC) orbiter warning: context deadline exceeded Duration: 3.449411429s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................................X...............X..X.......] duration: 34.293377263s trace: https://jaeger.dev.storj.io/trace/7e7354d8023608a8 FAIL (2025-12-29 11:28:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== ) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480097317 (571h53m39.351180352s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480405053 (571h53m39.351488098s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480712969 (571h53m39.351796014s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481019453 (571h53m39.352102488s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481029532 (571h53m39.352112567s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481559832 (571h53m39.352642877s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481731919 (571h53m39.352814954s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482078329 (571h53m39.353161364s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482099830 (571h53m39.353182865s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482434487 (571h53m39.353517532s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482480845 (571h53m39.35356388s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482841442 (571h53m39.353924477s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483163645 (571h53m39.35424668s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483532478 (571h53m39.354615513s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483914596 (571h53m39.354997631s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:40.761721106 (571h53m40.632804141s) [upload=501384]: Next returned nil segment Dec 29 11:22:40.761733349 (571h53m40.632816384s) [upload=501384]: Waiting for error group managing segments... Dec 29 11:22:41.121222862 (571h53m40.992305897s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.197311646 (571h53m41.068394691s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347247568 (571h53m41.218330613s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347917313 (571h53m41.219000348s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.348510613 (571h53m41.219593648s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.503950985 (571h53m41.37503402s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.561352025 (571h53m41.43243506s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.622447380 (571h53m41.493530425s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.645494685 (571h53m41.51657773s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.247930045 (571h53m42.11901308s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.262221890 (571h53m42.133304925s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.295598123 (571h53m42.166681158s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.307481351 (571h53m42.178564386s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325775906 (571h53m42.196858941s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325865978 (571h53m42.196949023s) [seg_pos={0 0}, upload=501384]: Segment reached optimal threshold of 14 pieces. Dec 29 11:22:42.326137074 (571h53m42.197220109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection Dec 29 11:22:42.326153074 (571h53m42.197236109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326193401 (571h53m42.197276446s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326205705 (571h53m42.19728874s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326407739 (571h53m42.197490784s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326415374 (571h53m42.197498419s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326505105 (571h53m42.19758815s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326512188 (571h53m42.197595223s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326532357 (571h53m42.197615392s) [seg_pos={0 0}, upload=501384]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 11:22:42.326671562 (571h53m42.197754597s) [upload=501384]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:22:42.326677843 (571h53m42.197760879s) [upload=501384]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:22:42.610391438 (571h53m42.481474493s) [upload=501384]: Done uploading segments... Dec 29 11:22:42.610416826 (571h53m42.481499871s) [upload=501384]: Upload finished. err: <nil> Duration: 34.293391601s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [XX..X.X.XX....X..X.....X.X...X.X.X.X.XX..XX..X..XXX.X.......] duration: 5.622972066s trace: https://jaeger.dev.storj.io/trace/478381ee752feccc FAIL (2025-12-29 11:28:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: HNWAPBAX36BBXIDGUPQDTDSQA4PZ4TGB22EUWV3RYGUHZCEZOE2Q) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: 4KU7BI4YDC62OWW452QQBBL6FPPQ4XFV6622H3S7NHVH3RC4QUNA) context deadline exceeded ======LOGS====== Dec 29 11:22:32.632698611 (571h53m32.503781656s) [upload=501383]: Starting upload Dec 29 11:22:32.632771780 (571h53m32.503854815s) [upload=501383]: Done starting upload Dec 29 11:22:32.632851863 (571h53m32.503934898s) [upload=501383]: Uploading segments... Dec 29 11:22:32.632999464 (571h53m32.504082509s) [upload=501383]: Got next segment. Inline: false Dec 29 11:22:32.633036064 (571h53m32.504119099s) [upload=501383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 11:22:32.633044419 (571h53m32.504127464s) [upload=501383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 11:22:32.881455231 (571h53m32.752538266s) [seg_pos={0 0}, upload=501383]: Begin upload segment... Dec 29 11:22:32.881502301 (571h53m32.752585336s) [seg_pos={0 0}, upload=501383]: Done begin upload segment. Dec 29 11:22:32.881527148 (571h53m32.752610183s) [noise=true , address=145.79.227.36:4519 , piece_id=CYHA5B5AAYLQYGG7PKDZBL6KBNL7PWAXC3XJESNZJZNG6VFGAUDQ, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.881534973 (571h53m32.752618008s) [noise=true , address=45.91.116.9:4264 , piece_id=O23BR37PP6JQXJVHRI65MNPHKXXBKEMXGEEKVKSFS3NJQIQEWX2Q, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.881630104 (571h53m32.752713139s) [noise=true , address=145.79.227.38:4978 , piece_id=WYIS25RQHPKBFK2T2NKYDLXXSRLTODHVGQESJS2EAGK76E7ATO3Q, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.882288178 (571h53m32.753371213s) [noise=true , address=145.79.227.3:4149 , piece_id=PO7SJV27DUHYSB73OAZ3OKAJYF4TVF4DWHT2HZAIC2A2YQVCKXXQ, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.882621682 (571h53m32.753704717s) [noise=true , address=77.90.31.10:4517 , piece_id=3QAS444K45QVWONIQJAJVQLKQPKD62ER2NUOFUAMUO7LCF7WWPGQ, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.883291408 (571h53m32.754374443s) [noise=true , address=45.91.116.8:4045 , piece_id=HNWAPBAX36BBXIDGUPQDTDSQA4PZ4TGB22EUWV3RYGUHZCEZOE2Q, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.883400165 (571h53m32.75448323s) [noise=true , address=45.91.116.10:4084 , piece_id=4KU7BI4YDC62OWW452QQBBL6FPPQ4XFV6622H3S7NHVH3RC4QUNA, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.883704695 (571h53m32.75478774s) [noise=true , address=145.79.227.16:4910 , piece_id=MR4IES6CULVSVQHYOYSNHUL6RWDIXQHES2S43HPMFLCKR726MCIA, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.884302744 (571h53m32.755385779s) [noise=true , address=145.79.227.50:4374 , piece_id=PYOD755FKX2UEJ2KGDSO77M4I5MY2GUGZ6WPKC5P73QYRQXTOYLQ, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.884750867 (571h53m32.755833912s) [noise=true , address=145.79.227.28:4726 , piece_id=H5FAH27QEOC6PMAZSX7FVEIMGI6JKTQAOYKMBXSFXRYNQT25324A, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.885400453 (571h53m32.756483489s) [noise=true , address=145.79.227.51:4409 , piece_id=3K5GWDCXZD2EU2L4J7J57IWCSGXHEMPFVO4YXH4SUCBGAFXOY7GA, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.886481201 (571h53m32.757564246s) [noise=true , address=45.91.116.14:4190 , piece_id=4LEIJANQSVBZH3XCRNKZ6LBQBK3P46Q6DOFD2VKFVCCBEIL7ECEA, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.887235819 (571h53m32.758318854s) [noise=true , address=145.79.227.19:4079 , piece_id=JCRBTXMT3G6GJA5K4XLZW2Y6KSHX5JAXW5GJGH6IMMWL5F47MNRQ, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.887920132 (571h53m32.759003167s) [noise=true , address=45.91.116.5:4297 , piece_id=VMKLHK4Z34NW75AVI4ZA2OA7ONSTY4C2RIR6T4AGANRCGHBIP4GA, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.889101662 (571h53m32.760184707s) [noise=true , address=145.79.227.37:4565 , piece_id=BSQN6OBVAEXEBBPDVSYAOQBG7ZNJUEMWSRIPSSIUZKVXWAI5QNEQ, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.889343362 (571h53m32.760426427s) [noise=true , address=145.79.227.6:4258 , piece_id=U6EG23QFBI3OR6ECAMRXZXZUWZ36RC5VY54LFXGWBEQN6PWNYBZA, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:34.576948050 (571h53m34.448031096s) [upload=501383]: Next returned nil segment Dec 29 11:22:34.576961886 (571h53m34.448044921s) [upload=501383]: Waiting for error group managing segments... Dec 29 11:22:34.861780225 (571h53m34.73286327s) [noise=true , address=145.79.227.3:4149 , piece_id=PO7SJV27DUHYSB73OAZ3OKAJYF4TVF4DWHT2HZAIC2A2YQVCKXXQ, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:34.862453887 (571h53m34.733536922s) [noise=true , address=145.79.227.37:4565 , piece_id=BSQN6OBVAEXEBBPDVSYAOQBG7ZNJUEMWSRIPSSIUZKVXWAI5QNEQ, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:34.863190951 (571h53m34.734273986s) [noise=true , address=145.79.227.38:4978 , piece_id=WYIS25RQHPKBFK2T2NKYDLXXSRLTODHVGQESJS2EAGK76E7ATO3Q, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:34.866655959 (571h53m34.737738994s) [noise=true , address=145.79.227.16:4910 , piece_id=MR4IES6CULVSVQHYOYSNHUL6RWDIXQHES2S43HPMFLCKR726MCIA, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:34.870192272 (571h53m34.741275317s) [noise=true , address=145.79.227.36:4519 , piece_id=CYHA5B5AAYLQYGG7PKDZBL6KBNL7PWAXC3XJESNZJZNG6VFGAUDQ, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:34.870875714 (571h53m34.741958749s) [noise=true , address=145.79.227.50:4374 , piece_id=PYOD755FKX2UEJ2KGDSO77M4I5MY2GUGZ6WPKC5P73QYRQXTOYLQ, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:34.872583666 (571h53m34.743666701s) [noise=true , address=145.79.227.28:4726 , piece_id=H5FAH27QEOC6PMAZSX7FVEIMGI6JKTQAOYKMBXSFXRYNQT25324A, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:34.890730350 (571h53m34.761813395s) [noise=true , address=145.79.227.51:4409 , piece_id=3K5GWDCXZD2EU2L4J7J57IWCSGXHEMPFVO4YXH4SUCBGAFXOY7GA, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:34.891395006 (571h53m34.762478051s) [noise=true , address=145.79.227.19:4079 , piece_id=JCRBTXMT3G6GJA5K4XLZW2Y6KSHX5JAXW5GJGH6IMMWL5F47MNRQ, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:34.895657111 (571h53m34.766740146s) [noise=true , address=145.79.227.6:4258 , piece_id=U6EG23QFBI3OR6ECAMRXZXZUWZ36RC5VY54LFXGWBEQN6PWNYBZA, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:35.227622103 (571h53m35.098705139s) [noise=true , address=45.91.116.5:4297 , piece_id=VMKLHK4Z34NW75AVI4ZA2OA7ONSTY4C2RIR6T4AGANRCGHBIP4GA, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:35.261404501 (571h53m35.132487556s) [noise=true , address=45.91.116.8:4045 , piece_id=HNWAPBAX36BBXIDGUPQDTDSQA4PZ4TGB22EUWV3RYGUHZCEZOE2Q, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:35.280892309 (571h53m35.151975344s) [noise=true , address=45.91.116.10:4084 , piece_id=4KU7BI4YDC62OWW452QQBBL6FPPQ4XFV6622H3S7NHVH3RC4QUNA, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:35.280914941 (571h53m35.151997986s) [seg_pos={0 0}, upload=501383]: Segment reached optimal threshold of 13 pieces. Dec 29 11:22:35.281224681 (571h53m35.152307746s) [noise=true , address=77.90.31.10:4517 , piece_id=3QAS444K45QVWONIQJAJVQLKQPKD62ER2NUOFUAMUO7LCF7WWPGQ, seg_pos={0 0}, upload=501383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:35.281234840 (571h53m35.152317885s) [noise=true , address=77.90.31.10:4517 , piece_id=3QAS444K45QVWONIQJAJVQLKQPKD62ER2NUOFUAMUO7LCF7WWPGQ, seg_pos={0 0}, upload=501383]: Long tail cancellation due to optimal threshold Dec 29 11:22:35.281401778 (571h53m35.152484823s) [noise=true , address=45.91.116.14:4190 , piece_id=4LEIJANQSVBZH3XCRNKZ6LBQBK3P46Q6DOFD2VKFVCCBEIL7ECEA, seg_pos={0 0}, upload=501383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:35.281424642 (571h53m35.152507677s) [noise=true , address=45.91.116.14:4190 , piece_id=4LEIJANQSVBZH3XCRNKZ6LBQBK3P46Q6DOFD2VKFVCCBEIL7ECEA, seg_pos={0 0}, upload=501383]: Long tail cancellation due to optimal threshold Dec 29 11:22:35.281569658 (571h53m35.152652703s) [noise=true , address=45.91.116.9:4264 , piece_id=O23BR37PP6JQXJVHRI65MNPHKXXBKEMXGEEKVKSFS3NJQIQEWX2Q, seg_pos={0 0}, upload=501383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:35.281578355 (571h53m35.15266139s) [noise=true , address=45.91.116.9:4264 , piece_id=O23BR37PP6JQXJVHRI65MNPHKXXBKEMXGEEKVKSFS3NJQIQEWX2Q, seg_pos={0 0}, upload=501383]: Long tail cancellation due to optimal threshold Dec 29 11:22:35.281598103 (571h53m35.152681148s) [seg_pos={0 0}, upload=501383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 11:22:35.281676261 (571h53m35.152759316s) [upload=501383]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:22:35.281682853 (571h53m35.152765889s) [upload=501383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:22:35.572843998 (571h53m35.443927044s) [upload=501383]: Done uploading segments... Dec 29 11:22:35.572864308 (571h53m35.443947353s) [upload=501383]: Upload finished. err: <nil> Duration: 5.622998495s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [.X...X....X...X..........X....X.......X..X...XX..XX......X.X] duration: 5.000990673s trace: https://jaeger.dev.storj.io/trace/76f9311d5d4a38d0 FAIL (2025-12-29 11:28:15 UTC) orbiter warning: context deadline exceeded Duration: 5.001014619s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................................X...............X..X......] duration: 42.425539448s trace: https://jaeger.dev.storj.io/trace/7e7354d8023608a8 FAIL (2025-12-29 11:27:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== ) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480097317 (571h53m39.351180352s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480405053 (571h53m39.351488098s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480712969 (571h53m39.351796014s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481019453 (571h53m39.352102488s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481029532 (571h53m39.352112567s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481559832 (571h53m39.352642877s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481731919 (571h53m39.352814954s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482078329 (571h53m39.353161364s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482099830 (571h53m39.353182865s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482434487 (571h53m39.353517532s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482480845 (571h53m39.35356388s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482841442 (571h53m39.353924477s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483163645 (571h53m39.35424668s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483532478 (571h53m39.354615513s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483914596 (571h53m39.354997631s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:40.761721106 (571h53m40.632804141s) [upload=501384]: Next returned nil segment Dec 29 11:22:40.761733349 (571h53m40.632816384s) [upload=501384]: Waiting for error group managing segments... Dec 29 11:22:41.121222862 (571h53m40.992305897s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.197311646 (571h53m41.068394691s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347247568 (571h53m41.218330613s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347917313 (571h53m41.219000348s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.348510613 (571h53m41.219593648s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.503950985 (571h53m41.37503402s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.561352025 (571h53m41.43243506s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.622447380 (571h53m41.493530425s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.645494685 (571h53m41.51657773s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.247930045 (571h53m42.11901308s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.262221890 (571h53m42.133304925s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.295598123 (571h53m42.166681158s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.307481351 (571h53m42.178564386s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325775906 (571h53m42.196858941s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325865978 (571h53m42.196949023s) [seg_pos={0 0}, upload=501384]: Segment reached optimal threshold of 14 pieces. Dec 29 11:22:42.326137074 (571h53m42.197220109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection Dec 29 11:22:42.326153074 (571h53m42.197236109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326193401 (571h53m42.197276446s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326205705 (571h53m42.19728874s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326407739 (571h53m42.197490784s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326415374 (571h53m42.197498419s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326505105 (571h53m42.19758815s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326512188 (571h53m42.197595223s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326532357 (571h53m42.197615392s) [seg_pos={0 0}, upload=501384]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 11:22:42.326671562 (571h53m42.197754597s) [upload=501384]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:22:42.326677843 (571h53m42.197760879s) [upload=501384]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:22:42.610391438 (571h53m42.481474493s) [upload=501384]: Done uploading segments... Dec 29 11:22:42.610416826 (571h53m42.481499871s) [upload=501384]: Upload finished. err: <nil> Duration: 42.425555629s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [XXX..X.X.XX....X..X.....X.X...X.X.X.X.XX..XX..X..XXX.X......] duration: 6.202611584s trace: https://jaeger.dev.storj.io/trace/478381ee752feccc FAIL (2025-12-29 11:27:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: HNWAPBAX36BBXIDGUPQDTDSQA4PZ4TGB22EUWV3RYGUHZCEZOE2Q) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: 4KU7BI4YDC62OWW452QQBBL6FPPQ4XFV6622H3S7NHVH3RC4QUNA) context deadline exceeded ======LOGS====== Dec 29 11:22:32.632698611 (571h53m32.503781656s) [upload=501383]: Starting upload Dec 29 11:22:32.632771780 (571h53m32.503854815s) [upload=501383]: Done starting upload Dec 29 11:22:32.632851863 (571h53m32.503934898s) [upload=501383]: Uploading segments... Dec 29 11:22:32.632999464 (571h53m32.504082509s) [upload=501383]: Got next segment. Inline: false Dec 29 11:22:32.633036064 (571h53m32.504119099s) [upload=501383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 11:22:32.633044419 (571h53m32.504127464s) [upload=501383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 11:22:32.881455231 (571h53m32.752538266s) [seg_pos={0 0}, upload=501383]: Begin upload segment... Dec 29 11:22:32.881502301 (571h53m32.752585336s) [seg_pos={0 0}, upload=501383]: Done begin upload segment. Dec 29 11:22:32.881527148 (571h53m32.752610183s) [noise=true , address=145.79.227.36:4519 , piece_id=CYHA5B5AAYLQYGG7PKDZBL6KBNL7PWAXC3XJESNZJZNG6VFGAUDQ, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.881534973 (571h53m32.752618008s) [noise=true , address=45.91.116.9:4264 , piece_id=O23BR37PP6JQXJVHRI65MNPHKXXBKEMXGEEKVKSFS3NJQIQEWX2Q, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.881630104 (571h53m32.752713139s) [noise=true , address=145.79.227.38:4978 , piece_id=WYIS25RQHPKBFK2T2NKYDLXXSRLTODHVGQESJS2EAGK76E7ATO3Q, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.882288178 (571h53m32.753371213s) [noise=true , address=145.79.227.3:4149 , piece_id=PO7SJV27DUHYSB73OAZ3OKAJYF4TVF4DWHT2HZAIC2A2YQVCKXXQ, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.882621682 (571h53m32.753704717s) [noise=true , address=77.90.31.10:4517 , piece_id=3QAS444K45QVWONIQJAJVQLKQPKD62ER2NUOFUAMUO7LCF7WWPGQ, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.883291408 (571h53m32.754374443s) [noise=true , address=45.91.116.8:4045 , piece_id=HNWAPBAX36BBXIDGUPQDTDSQA4PZ4TGB22EUWV3RYGUHZCEZOE2Q, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.883400165 (571h53m32.75448323s) [noise=true , address=45.91.116.10:4084 , piece_id=4KU7BI4YDC62OWW452QQBBL6FPPQ4XFV6622H3S7NHVH3RC4QUNA, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.883704695 (571h53m32.75478774s) [noise=true , address=145.79.227.16:4910 , piece_id=MR4IES6CULVSVQHYOYSNHUL6RWDIXQHES2S43HPMFLCKR726MCIA, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.884302744 (571h53m32.755385779s) [noise=true , address=145.79.227.50:4374 , piece_id=PYOD755FKX2UEJ2KGDSO77M4I5MY2GUGZ6WPKC5P73QYRQXTOYLQ, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.884750867 (571h53m32.755833912s) [noise=true , address=145.79.227.28:4726 , piece_id=H5FAH27QEOC6PMAZSX7FVEIMGI6JKTQAOYKMBXSFXRYNQT25324A, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.885400453 (571h53m32.756483489s) [noise=true , address=145.79.227.51:4409 , piece_id=3K5GWDCXZD2EU2L4J7J57IWCSGXHEMPFVO4YXH4SUCBGAFXOY7GA, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.886481201 (571h53m32.757564246s) [noise=true , address=45.91.116.14:4190 , piece_id=4LEIJANQSVBZH3XCRNKZ6LBQBK3P46Q6DOFD2VKFVCCBEIL7ECEA, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.887235819 (571h53m32.758318854s) [noise=true , address=145.79.227.19:4079 , piece_id=JCRBTXMT3G6GJA5K4XLZW2Y6KSHX5JAXW5GJGH6IMMWL5F47MNRQ, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.887920132 (571h53m32.759003167s) [noise=true , address=45.91.116.5:4297 , piece_id=VMKLHK4Z34NW75AVI4ZA2OA7ONSTY4C2RIR6T4AGANRCGHBIP4GA, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.889101662 (571h53m32.760184707s) [noise=true , address=145.79.227.37:4565 , piece_id=BSQN6OBVAEXEBBPDVSYAOQBG7ZNJUEMWSRIPSSIUZKVXWAI5QNEQ, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.889343362 (571h53m32.760426427s) [noise=true , address=145.79.227.6:4258 , piece_id=U6EG23QFBI3OR6ECAMRXZXZUWZ36RC5VY54LFXGWBEQN6PWNYBZA, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:34.576948050 (571h53m34.448031096s) [upload=501383]: Next returned nil segment Dec 29 11:22:34.576961886 (571h53m34.448044921s) [upload=501383]: Waiting for error group managing segments... Dec 29 11:22:34.861780225 (571h53m34.73286327s) [noise=true , address=145.79.227.3:4149 , piece_id=PO7SJV27DUHYSB73OAZ3OKAJYF4TVF4DWHT2HZAIC2A2YQVCKXXQ, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:34.862453887 (571h53m34.733536922s) [noise=true , address=145.79.227.37:4565 , piece_id=BSQN6OBVAEXEBBPDVSYAOQBG7ZNJUEMWSRIPSSIUZKVXWAI5QNEQ, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:34.863190951 (571h53m34.734273986s) [noise=true , address=145.79.227.38:4978 , piece_id=WYIS25RQHPKBFK2T2NKYDLXXSRLTODHVGQESJS2EAGK76E7ATO3Q, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:34.866655959 (571h53m34.737738994s) [noise=true , address=145.79.227.16:4910 , piece_id=MR4IES6CULVSVQHYOYSNHUL6RWDIXQHES2S43HPMFLCKR726MCIA, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:34.870192272 (571h53m34.741275317s) [noise=true , address=145.79.227.36:4519 , piece_id=CYHA5B5AAYLQYGG7PKDZBL6KBNL7PWAXC3XJESNZJZNG6VFGAUDQ, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:34.870875714 (571h53m34.741958749s) [noise=true , address=145.79.227.50:4374 , piece_id=PYOD755FKX2UEJ2KGDSO77M4I5MY2GUGZ6WPKC5P73QYRQXTOYLQ, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:34.872583666 (571h53m34.743666701s) [noise=true , address=145.79.227.28:4726 , piece_id=H5FAH27QEOC6PMAZSX7FVEIMGI6JKTQAOYKMBXSFXRYNQT25324A, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:34.890730350 (571h53m34.761813395s) [noise=true , address=145.79.227.51:4409 , piece_id=3K5GWDCXZD2EU2L4J7J57IWCSGXHEMPFVO4YXH4SUCBGAFXOY7GA, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:34.891395006 (571h53m34.762478051s) [noise=true , address=145.79.227.19:4079 , piece_id=JCRBTXMT3G6GJA5K4XLZW2Y6KSHX5JAXW5GJGH6IMMWL5F47MNRQ, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:34.895657111 (571h53m34.766740146s) [noise=true , address=145.79.227.6:4258 , piece_id=U6EG23QFBI3OR6ECAMRXZXZUWZ36RC5VY54LFXGWBEQN6PWNYBZA, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:35.227622103 (571h53m35.098705139s) [noise=true , address=45.91.116.5:4297 , piece_id=VMKLHK4Z34NW75AVI4ZA2OA7ONSTY4C2RIR6T4AGANRCGHBIP4GA, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:35.261404501 (571h53m35.132487556s) [noise=true , address=45.91.116.8:4045 , piece_id=HNWAPBAX36BBXIDGUPQDTDSQA4PZ4TGB22EUWV3RYGUHZCEZOE2Q, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:35.280892309 (571h53m35.151975344s) [noise=true , address=45.91.116.10:4084 , piece_id=4KU7BI4YDC62OWW452QQBBL6FPPQ4XFV6622H3S7NHVH3RC4QUNA, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:35.280914941 (571h53m35.151997986s) [seg_pos={0 0}, upload=501383]: Segment reached optimal threshold of 13 pieces. Dec 29 11:22:35.281224681 (571h53m35.152307746s) [noise=true , address=77.90.31.10:4517 , piece_id=3QAS444K45QVWONIQJAJVQLKQPKD62ER2NUOFUAMUO7LCF7WWPGQ, seg_pos={0 0}, upload=501383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:35.281234840 (571h53m35.152317885s) [noise=true , address=77.90.31.10:4517 , piece_id=3QAS444K45QVWONIQJAJVQLKQPKD62ER2NUOFUAMUO7LCF7WWPGQ, seg_pos={0 0}, upload=501383]: Long tail cancellation due to optimal threshold Dec 29 11:22:35.281401778 (571h53m35.152484823s) [noise=true , address=45.91.116.14:4190 , piece_id=4LEIJANQSVBZH3XCRNKZ6LBQBK3P46Q6DOFD2VKFVCCBEIL7ECEA, seg_pos={0 0}, upload=501383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:35.281424642 (571h53m35.152507677s) [noise=true , address=45.91.116.14:4190 , piece_id=4LEIJANQSVBZH3XCRNKZ6LBQBK3P46Q6DOFD2VKFVCCBEIL7ECEA, seg_pos={0 0}, upload=501383]: Long tail cancellation due to optimal threshold Dec 29 11:22:35.281569658 (571h53m35.152652703s) [noise=true , address=45.91.116.9:4264 , piece_id=O23BR37PP6JQXJVHRI65MNPHKXXBKEMXGEEKVKSFS3NJQIQEWX2Q, seg_pos={0 0}, upload=501383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:35.281578355 (571h53m35.15266139s) [noise=true , address=45.91.116.9:4264 , piece_id=O23BR37PP6JQXJVHRI65MNPHKXXBKEMXGEEKVKSFS3NJQIQEWX2Q, seg_pos={0 0}, upload=501383]: Long tail cancellation due to optimal threshold Dec 29 11:22:35.281598103 (571h53m35.152681148s) [seg_pos={0 0}, upload=501383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 11:22:35.281676261 (571h53m35.152759316s) [upload=501383]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:22:35.281682853 (571h53m35.152765889s) [upload=501383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:22:35.572843998 (571h53m35.443927044s) [upload=501383]: Done uploading segments... Dec 29 11:22:35.572864308 (571h53m35.443947353s) [upload=501383]: Upload finished. err: <nil> Duration: 6.202639478s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.XXX..X.X.XX....X..X.....X.X...X.X.X.X.XX..XX..X..XXX.X.....] duration: 5.698256555s trace: https://jaeger.dev.storj.io/trace/478381ee752feccc FAIL (2025-12-29 11:27:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: HNWAPBAX36BBXIDGUPQDTDSQA4PZ4TGB22EUWV3RYGUHZCEZOE2Q) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: 4KU7BI4YDC62OWW452QQBBL6FPPQ4XFV6622H3S7NHVH3RC4QUNA) context deadline exceeded ======LOGS====== Dec 29 11:22:32.632698611 (571h53m32.503781656s) [upload=501383]: Starting upload Dec 29 11:22:32.632771780 (571h53m32.503854815s) [upload=501383]: Done starting upload Dec 29 11:22:32.632851863 (571h53m32.503934898s) [upload=501383]: Uploading segments... Dec 29 11:22:32.632999464 (571h53m32.504082509s) [upload=501383]: Got next segment. Inline: false Dec 29 11:22:32.633036064 (571h53m32.504119099s) [upload=501383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 11:22:32.633044419 (571h53m32.504127464s) [upload=501383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 11:22:32.881455231 (571h53m32.752538266s) [seg_pos={0 0}, upload=501383]: Begin upload segment... Dec 29 11:22:32.881502301 (571h53m32.752585336s) [seg_pos={0 0}, upload=501383]: Done begin upload segment. Dec 29 11:22:32.881527148 (571h53m32.752610183s) [noise=true , address=145.79.227.36:4519 , piece_id=CYHA5B5AAYLQYGG7PKDZBL6KBNL7PWAXC3XJESNZJZNG6VFGAUDQ, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.881534973 (571h53m32.752618008s) [noise=true , address=45.91.116.9:4264 , piece_id=O23BR37PP6JQXJVHRI65MNPHKXXBKEMXGEEKVKSFS3NJQIQEWX2Q, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.881630104 (571h53m32.752713139s) [noise=true , address=145.79.227.38:4978 , piece_id=WYIS25RQHPKBFK2T2NKYDLXXSRLTODHVGQESJS2EAGK76E7ATO3Q, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.882288178 (571h53m32.753371213s) [noise=true , address=145.79.227.3:4149 , piece_id=PO7SJV27DUHYSB73OAZ3OKAJYF4TVF4DWHT2HZAIC2A2YQVCKXXQ, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.882621682 (571h53m32.753704717s) [noise=true , address=77.90.31.10:4517 , piece_id=3QAS444K45QVWONIQJAJVQLKQPKD62ER2NUOFUAMUO7LCF7WWPGQ, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.883291408 (571h53m32.754374443s) [noise=true , address=45.91.116.8:4045 , piece_id=HNWAPBAX36BBXIDGUPQDTDSQA4PZ4TGB22EUWV3RYGUHZCEZOE2Q, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.883400165 (571h53m32.75448323s) [noise=true , address=45.91.116.10:4084 , piece_id=4KU7BI4YDC62OWW452QQBBL6FPPQ4XFV6622H3S7NHVH3RC4QUNA, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.883704695 (571h53m32.75478774s) [noise=true , address=145.79.227.16:4910 , piece_id=MR4IES6CULVSVQHYOYSNHUL6RWDIXQHES2S43HPMFLCKR726MCIA, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.884302744 (571h53m32.755385779s) [noise=true , address=145.79.227.50:4374 , piece_id=PYOD755FKX2UEJ2KGDSO77M4I5MY2GUGZ6WPKC5P73QYRQXTOYLQ, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.884750867 (571h53m32.755833912s) [noise=true , address=145.79.227.28:4726 , piece_id=H5FAH27QEOC6PMAZSX7FVEIMGI6JKTQAOYKMBXSFXRYNQT25324A, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.885400453 (571h53m32.756483489s) [noise=true , address=145.79.227.51:4409 , piece_id=3K5GWDCXZD2EU2L4J7J57IWCSGXHEMPFVO4YXH4SUCBGAFXOY7GA, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.886481201 (571h53m32.757564246s) [noise=true , address=45.91.116.14:4190 , piece_id=4LEIJANQSVBZH3XCRNKZ6LBQBK3P46Q6DOFD2VKFVCCBEIL7ECEA, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.887235819 (571h53m32.758318854s) [noise=true , address=145.79.227.19:4079 , piece_id=JCRBTXMT3G6GJA5K4XLZW2Y6KSHX5JAXW5GJGH6IMMWL5F47MNRQ, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.887920132 (571h53m32.759003167s) [noise=true , address=45.91.116.5:4297 , piece_id=VMKLHK4Z34NW75AVI4ZA2OA7ONSTY4C2RIR6T4AGANRCGHBIP4GA, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.889101662 (571h53m32.760184707s) [noise=true , address=145.79.227.37:4565 , piece_id=BSQN6OBVAEXEBBPDVSYAOQBG7ZNJUEMWSRIPSSIUZKVXWAI5QNEQ, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.889343362 (571h53m32.760426427s) [noise=true , address=145.79.227.6:4258 , piece_id=U6EG23QFBI3OR6ECAMRXZXZUWZ36RC5VY54LFXGWBEQN6PWNYBZA, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:34.576948050 (571h53m34.448031096s) [upload=501383]: Next returned nil segment Dec 29 11:22:34.576961886 (571h53m34.448044921s) [upload=501383]: Waiting for error group managing segments... Dec 29 11:22:34.861780225 (571h53m34.73286327s) [noise=true , address=145.79.227.3:4149 , piece_id=PO7SJV27DUHYSB73OAZ3OKAJYF4TVF4DWHT2HZAIC2A2YQVCKXXQ, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:34.862453887 (571h53m34.733536922s) [noise=true , address=145.79.227.37:4565 , piece_id=BSQN6OBVAEXEBBPDVSYAOQBG7ZNJUEMWSRIPSSIUZKVXWAI5QNEQ, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:34.863190951 (571h53m34.734273986s) [noise=true , address=145.79.227.38:4978 , piece_id=WYIS25RQHPKBFK2T2NKYDLXXSRLTODHVGQESJS2EAGK76E7ATO3Q, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:34.866655959 (571h53m34.737738994s) [noise=true , address=145.79.227.16:4910 , piece_id=MR4IES6CULVSVQHYOYSNHUL6RWDIXQHES2S43HPMFLCKR726MCIA, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:34.870192272 (571h53m34.741275317s) [noise=true , address=145.79.227.36:4519 , piece_id=CYHA5B5AAYLQYGG7PKDZBL6KBNL7PWAXC3XJESNZJZNG6VFGAUDQ, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:34.870875714 (571h53m34.741958749s) [noise=true , address=145.79.227.50:4374 , piece_id=PYOD755FKX2UEJ2KGDSO77M4I5MY2GUGZ6WPKC5P73QYRQXTOYLQ, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:34.872583666 (571h53m34.743666701s) [noise=true , address=145.79.227.28:4726 , piece_id=H5FAH27QEOC6PMAZSX7FVEIMGI6JKTQAOYKMBXSFXRYNQT25324A, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:34.890730350 (571h53m34.761813395s) [noise=true , address=145.79.227.51:4409 , piece_id=3K5GWDCXZD2EU2L4J7J57IWCSGXHEMPFVO4YXH4SUCBGAFXOY7GA, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:34.891395006 (571h53m34.762478051s) [noise=true , address=145.79.227.19:4079 , piece_id=JCRBTXMT3G6GJA5K4XLZW2Y6KSHX5JAXW5GJGH6IMMWL5F47MNRQ, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:34.895657111 (571h53m34.766740146s) [noise=true , address=145.79.227.6:4258 , piece_id=U6EG23QFBI3OR6ECAMRXZXZUWZ36RC5VY54LFXGWBEQN6PWNYBZA, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:35.227622103 (571h53m35.098705139s) [noise=true , address=45.91.116.5:4297 , piece_id=VMKLHK4Z34NW75AVI4ZA2OA7ONSTY4C2RIR6T4AGANRCGHBIP4GA, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:35.261404501 (571h53m35.132487556s) [noise=true , address=45.91.116.8:4045 , piece_id=HNWAPBAX36BBXIDGUPQDTDSQA4PZ4TGB22EUWV3RYGUHZCEZOE2Q, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:35.280892309 (571h53m35.151975344s) [noise=true , address=45.91.116.10:4084 , piece_id=4KU7BI4YDC62OWW452QQBBL6FPPQ4XFV6622H3S7NHVH3RC4QUNA, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:35.280914941 (571h53m35.151997986s) [seg_pos={0 0}, upload=501383]: Segment reached optimal threshold of 13 pieces. Dec 29 11:22:35.281224681 (571h53m35.152307746s) [noise=true , address=77.90.31.10:4517 , piece_id=3QAS444K45QVWONIQJAJVQLKQPKD62ER2NUOFUAMUO7LCF7WWPGQ, seg_pos={0 0}, upload=501383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:35.281234840 (571h53m35.152317885s) [noise=true , address=77.90.31.10:4517 , piece_id=3QAS444K45QVWONIQJAJVQLKQPKD62ER2NUOFUAMUO7LCF7WWPGQ, seg_pos={0 0}, upload=501383]: Long tail cancellation due to optimal threshold Dec 29 11:22:35.281401778 (571h53m35.152484823s) [noise=true , address=45.91.116.14:4190 , piece_id=4LEIJANQSVBZH3XCRNKZ6LBQBK3P46Q6DOFD2VKFVCCBEIL7ECEA, seg_pos={0 0}, upload=501383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:35.281424642 (571h53m35.152507677s) [noise=true , address=45.91.116.14:4190 , piece_id=4LEIJANQSVBZH3XCRNKZ6LBQBK3P46Q6DOFD2VKFVCCBEIL7ECEA, seg_pos={0 0}, upload=501383]: Long tail cancellation due to optimal threshold Dec 29 11:22:35.281569658 (571h53m35.152652703s) [noise=true , address=45.91.116.9:4264 , piece_id=O23BR37PP6JQXJVHRI65MNPHKXXBKEMXGEEKVKSFS3NJQIQEWX2Q, seg_pos={0 0}, upload=501383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:35.281578355 (571h53m35.15266139s) [noise=true , address=45.91.116.9:4264 , piece_id=O23BR37PP6JQXJVHRI65MNPHKXXBKEMXGEEKVKSFS3NJQIQEWX2Q, seg_pos={0 0}, upload=501383]: Long tail cancellation due to optimal threshold Dec 29 11:22:35.281598103 (571h53m35.152681148s) [seg_pos={0 0}, upload=501383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 11:22:35.281676261 (571h53m35.152759316s) [upload=501383]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:22:35.281682853 (571h53m35.152765889s) [upload=501383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:22:35.572843998 (571h53m35.443927044s) [upload=501383]: Done uploading segments... Dec 29 11:22:35.572864308 (571h53m35.443947353s) [upload=501383]: Upload finished. err: <nil> Duration: 5.698282927s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [X..................................X...............X..X.....] duration: 38.418991599s trace: https://jaeger.dev.storj.io/trace/7e7354d8023608a8 FAIL (2025-12-29 11:27:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== ) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480097317 (571h53m39.351180352s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480405053 (571h53m39.351488098s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480712969 (571h53m39.351796014s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481019453 (571h53m39.352102488s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481029532 (571h53m39.352112567s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481559832 (571h53m39.352642877s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481731919 (571h53m39.352814954s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482078329 (571h53m39.353161364s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482099830 (571h53m39.353182865s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482434487 (571h53m39.353517532s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482480845 (571h53m39.35356388s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482841442 (571h53m39.353924477s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483163645 (571h53m39.35424668s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483532478 (571h53m39.354615513s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483914596 (571h53m39.354997631s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:40.761721106 (571h53m40.632804141s) [upload=501384]: Next returned nil segment Dec 29 11:22:40.761733349 (571h53m40.632816384s) [upload=501384]: Waiting for error group managing segments... Dec 29 11:22:41.121222862 (571h53m40.992305897s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.197311646 (571h53m41.068394691s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347247568 (571h53m41.218330613s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347917313 (571h53m41.219000348s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.348510613 (571h53m41.219593648s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.503950985 (571h53m41.37503402s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.561352025 (571h53m41.43243506s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.622447380 (571h53m41.493530425s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.645494685 (571h53m41.51657773s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.247930045 (571h53m42.11901308s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.262221890 (571h53m42.133304925s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.295598123 (571h53m42.166681158s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.307481351 (571h53m42.178564386s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325775906 (571h53m42.196858941s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325865978 (571h53m42.196949023s) [seg_pos={0 0}, upload=501384]: Segment reached optimal threshold of 14 pieces. Dec 29 11:22:42.326137074 (571h53m42.197220109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection Dec 29 11:22:42.326153074 (571h53m42.197236109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326193401 (571h53m42.197276446s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326205705 (571h53m42.19728874s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326407739 (571h53m42.197490784s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326415374 (571h53m42.197498419s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326505105 (571h53m42.19758815s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326512188 (571h53m42.197595223s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326532357 (571h53m42.197615392s) [seg_pos={0 0}, upload=501384]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 11:22:42.326671562 (571h53m42.197754597s) [upload=501384]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:22:42.326677843 (571h53m42.197760879s) [upload=501384]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:22:42.610391438 (571h53m42.481474493s) [upload=501384]: Done uploading segments... Dec 29 11:22:42.610416826 (571h53m42.481499871s) [upload=501384]: Upload finished. err: <nil> Duration: 38.419019152s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [..X...X....X...X..........X....X.......X..X...XX..XX......X.] duration: 5.000557679s trace: https://jaeger.dev.storj.io/trace/71610be972610fcb FAIL (2025-12-29 11:27:07 UTC) orbiter warning: context deadline exceeded Duration: 5.000621582s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [...X...X....X...X..........X....X.......X..X...XX..XX......X] duration: 5.000852799s trace: https://jaeger.dev.storj.io/trace/71610be972610fcb FAIL (2025-12-29 11:26:38 UTC) orbiter warning: context deadline exceeded Duration: 5.000873649s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [..XXX..X.X.XX....X..X.....X.X...X.X.X.X.XX..XX..X..XXX.X....] duration: 5.935837985s trace: https://jaeger.dev.storj.io/trace/478381ee752feccc FAIL (2025-12-29 11:26:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: HNWAPBAX36BBXIDGUPQDTDSQA4PZ4TGB22EUWV3RYGUHZCEZOE2Q) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: 4KU7BI4YDC62OWW452QQBBL6FPPQ4XFV6622H3S7NHVH3RC4QUNA) context deadline exceeded ======LOGS====== Dec 29 11:22:32.632698611 (571h53m32.503781656s) [upload=501383]: Starting upload Dec 29 11:22:32.632771780 (571h53m32.503854815s) [upload=501383]: Done starting upload Dec 29 11:22:32.632851863 (571h53m32.503934898s) [upload=501383]: Uploading segments... Dec 29 11:22:32.632999464 (571h53m32.504082509s) [upload=501383]: Got next segment. Inline: false Dec 29 11:22:32.633036064 (571h53m32.504119099s) [upload=501383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 11:22:32.633044419 (571h53m32.504127464s) [upload=501383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 11:22:32.881455231 (571h53m32.752538266s) [seg_pos={0 0}, upload=501383]: Begin upload segment... Dec 29 11:22:32.881502301 (571h53m32.752585336s) [seg_pos={0 0}, upload=501383]: Done begin upload segment. Dec 29 11:22:32.881527148 (571h53m32.752610183s) [noise=true , address=145.79.227.36:4519 , piece_id=CYHA5B5AAYLQYGG7PKDZBL6KBNL7PWAXC3XJESNZJZNG6VFGAUDQ, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.881534973 (571h53m32.752618008s) [noise=true , address=45.91.116.9:4264 , piece_id=O23BR37PP6JQXJVHRI65MNPHKXXBKEMXGEEKVKSFS3NJQIQEWX2Q, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.881630104 (571h53m32.752713139s) [noise=true , address=145.79.227.38:4978 , piece_id=WYIS25RQHPKBFK2T2NKYDLXXSRLTODHVGQESJS2EAGK76E7ATO3Q, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.882288178 (571h53m32.753371213s) [noise=true , address=145.79.227.3:4149 , piece_id=PO7SJV27DUHYSB73OAZ3OKAJYF4TVF4DWHT2HZAIC2A2YQVCKXXQ, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.882621682 (571h53m32.753704717s) [noise=true , address=77.90.31.10:4517 , piece_id=3QAS444K45QVWONIQJAJVQLKQPKD62ER2NUOFUAMUO7LCF7WWPGQ, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.883291408 (571h53m32.754374443s) [noise=true , address=45.91.116.8:4045 , piece_id=HNWAPBAX36BBXIDGUPQDTDSQA4PZ4TGB22EUWV3RYGUHZCEZOE2Q, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.883400165 (571h53m32.75448323s) [noise=true , address=45.91.116.10:4084 , piece_id=4KU7BI4YDC62OWW452QQBBL6FPPQ4XFV6622H3S7NHVH3RC4QUNA, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.883704695 (571h53m32.75478774s) [noise=true , address=145.79.227.16:4910 , piece_id=MR4IES6CULVSVQHYOYSNHUL6RWDIXQHES2S43HPMFLCKR726MCIA, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.884302744 (571h53m32.755385779s) [noise=true , address=145.79.227.50:4374 , piece_id=PYOD755FKX2UEJ2KGDSO77M4I5MY2GUGZ6WPKC5P73QYRQXTOYLQ, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.884750867 (571h53m32.755833912s) [noise=true , address=145.79.227.28:4726 , piece_id=H5FAH27QEOC6PMAZSX7FVEIMGI6JKTQAOYKMBXSFXRYNQT25324A, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.885400453 (571h53m32.756483489s) [noise=true , address=145.79.227.51:4409 , piece_id=3K5GWDCXZD2EU2L4J7J57IWCSGXHEMPFVO4YXH4SUCBGAFXOY7GA, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.886481201 (571h53m32.757564246s) [noise=true , address=45.91.116.14:4190 , piece_id=4LEIJANQSVBZH3XCRNKZ6LBQBK3P46Q6DOFD2VKFVCCBEIL7ECEA, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.887235819 (571h53m32.758318854s) [noise=true , address=145.79.227.19:4079 , piece_id=JCRBTXMT3G6GJA5K4XLZW2Y6KSHX5JAXW5GJGH6IMMWL5F47MNRQ, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.887920132 (571h53m32.759003167s) [noise=true , address=45.91.116.5:4297 , piece_id=VMKLHK4Z34NW75AVI4ZA2OA7ONSTY4C2RIR6T4AGANRCGHBIP4GA, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.889101662 (571h53m32.760184707s) [noise=true , address=145.79.227.37:4565 , piece_id=BSQN6OBVAEXEBBPDVSYAOQBG7ZNJUEMWSRIPSSIUZKVXWAI5QNEQ, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.889343362 (571h53m32.760426427s) [noise=true , address=145.79.227.6:4258 , piece_id=U6EG23QFBI3OR6ECAMRXZXZUWZ36RC5VY54LFXGWBEQN6PWNYBZA, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:34.576948050 (571h53m34.448031096s) [upload=501383]: Next returned nil segment Dec 29 11:22:34.576961886 (571h53m34.448044921s) [upload=501383]: Waiting for error group managing segments... Dec 29 11:22:34.861780225 (571h53m34.73286327s) [noise=true , address=145.79.227.3:4149 , piece_id=PO7SJV27DUHYSB73OAZ3OKAJYF4TVF4DWHT2HZAIC2A2YQVCKXXQ, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:34.862453887 (571h53m34.733536922s) [noise=true , address=145.79.227.37:4565 , piece_id=BSQN6OBVAEXEBBPDVSYAOQBG7ZNJUEMWSRIPSSIUZKVXWAI5QNEQ, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:34.863190951 (571h53m34.734273986s) [noise=true , address=145.79.227.38:4978 , piece_id=WYIS25RQHPKBFK2T2NKYDLXXSRLTODHVGQESJS2EAGK76E7ATO3Q, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:34.866655959 (571h53m34.737738994s) [noise=true , address=145.79.227.16:4910 , piece_id=MR4IES6CULVSVQHYOYSNHUL6RWDIXQHES2S43HPMFLCKR726MCIA, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:34.870192272 (571h53m34.741275317s) [noise=true , address=145.79.227.36:4519 , piece_id=CYHA5B5AAYLQYGG7PKDZBL6KBNL7PWAXC3XJESNZJZNG6VFGAUDQ, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:34.870875714 (571h53m34.741958749s) [noise=true , address=145.79.227.50:4374 , piece_id=PYOD755FKX2UEJ2KGDSO77M4I5MY2GUGZ6WPKC5P73QYRQXTOYLQ, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:34.872583666 (571h53m34.743666701s) [noise=true , address=145.79.227.28:4726 , piece_id=H5FAH27QEOC6PMAZSX7FVEIMGI6JKTQAOYKMBXSFXRYNQT25324A, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:34.890730350 (571h53m34.761813395s) [noise=true , address=145.79.227.51:4409 , piece_id=3K5GWDCXZD2EU2L4J7J57IWCSGXHEMPFVO4YXH4SUCBGAFXOY7GA, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:34.891395006 (571h53m34.762478051s) [noise=true , address=145.79.227.19:4079 , piece_id=JCRBTXMT3G6GJA5K4XLZW2Y6KSHX5JAXW5GJGH6IMMWL5F47MNRQ, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:34.895657111 (571h53m34.766740146s) [noise=true , address=145.79.227.6:4258 , piece_id=U6EG23QFBI3OR6ECAMRXZXZUWZ36RC5VY54LFXGWBEQN6PWNYBZA, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:35.227622103 (571h53m35.098705139s) [noise=true , address=45.91.116.5:4297 , piece_id=VMKLHK4Z34NW75AVI4ZA2OA7ONSTY4C2RIR6T4AGANRCGHBIP4GA, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:35.261404501 (571h53m35.132487556s) [noise=true , address=45.91.116.8:4045 , piece_id=HNWAPBAX36BBXIDGUPQDTDSQA4PZ4TGB22EUWV3RYGUHZCEZOE2Q, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:35.280892309 (571h53m35.151975344s) [noise=true , address=45.91.116.10:4084 , piece_id=4KU7BI4YDC62OWW452QQBBL6FPPQ4XFV6622H3S7NHVH3RC4QUNA, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:35.280914941 (571h53m35.151997986s) [seg_pos={0 0}, upload=501383]: Segment reached optimal threshold of 13 pieces. Dec 29 11:22:35.281224681 (571h53m35.152307746s) [noise=true , address=77.90.31.10:4517 , piece_id=3QAS444K45QVWONIQJAJVQLKQPKD62ER2NUOFUAMUO7LCF7WWPGQ, seg_pos={0 0}, upload=501383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:35.281234840 (571h53m35.152317885s) [noise=true , address=77.90.31.10:4517 , piece_id=3QAS444K45QVWONIQJAJVQLKQPKD62ER2NUOFUAMUO7LCF7WWPGQ, seg_pos={0 0}, upload=501383]: Long tail cancellation due to optimal threshold Dec 29 11:22:35.281401778 (571h53m35.152484823s) [noise=true , address=45.91.116.14:4190 , piece_id=4LEIJANQSVBZH3XCRNKZ6LBQBK3P46Q6DOFD2VKFVCCBEIL7ECEA, seg_pos={0 0}, upload=501383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:35.281424642 (571h53m35.152507677s) [noise=true , address=45.91.116.14:4190 , piece_id=4LEIJANQSVBZH3XCRNKZ6LBQBK3P46Q6DOFD2VKFVCCBEIL7ECEA, seg_pos={0 0}, upload=501383]: Long tail cancellation due to optimal threshold Dec 29 11:22:35.281569658 (571h53m35.152652703s) [noise=true , address=45.91.116.9:4264 , piece_id=O23BR37PP6JQXJVHRI65MNPHKXXBKEMXGEEKVKSFS3NJQIQEWX2Q, seg_pos={0 0}, upload=501383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:35.281578355 (571h53m35.15266139s) [noise=true , address=45.91.116.9:4264 , piece_id=O23BR37PP6JQXJVHRI65MNPHKXXBKEMXGEEKVKSFS3NJQIQEWX2Q, seg_pos={0 0}, upload=501383]: Long tail cancellation due to optimal threshold Dec 29 11:22:35.281598103 (571h53m35.152681148s) [seg_pos={0 0}, upload=501383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 11:22:35.281676261 (571h53m35.152759316s) [upload=501383]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:22:35.281682853 (571h53m35.152765889s) [upload=501383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:22:35.572843998 (571h53m35.443927044s) [upload=501383]: Done uploading segments... Dec 29 11:22:35.572864308 (571h53m35.443947353s) [upload=501383]: Upload finished. err: <nil> Duration: 5.935860478s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X..................................X...............X..X....] duration: 39.368736672s trace: https://jaeger.dev.storj.io/trace/7e7354d8023608a8 FAIL (2025-12-29 11:26:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== ) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480097317 (571h53m39.351180352s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480405053 (571h53m39.351488098s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480712969 (571h53m39.351796014s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481019453 (571h53m39.352102488s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481029532 (571h53m39.352112567s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481559832 (571h53m39.352642877s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481731919 (571h53m39.352814954s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482078329 (571h53m39.353161364s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482099830 (571h53m39.353182865s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482434487 (571h53m39.353517532s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482480845 (571h53m39.35356388s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482841442 (571h53m39.353924477s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483163645 (571h53m39.35424668s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483532478 (571h53m39.354615513s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483914596 (571h53m39.354997631s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:40.761721106 (571h53m40.632804141s) [upload=501384]: Next returned nil segment Dec 29 11:22:40.761733349 (571h53m40.632816384s) [upload=501384]: Waiting for error group managing segments... Dec 29 11:22:41.121222862 (571h53m40.992305897s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.197311646 (571h53m41.068394691s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347247568 (571h53m41.218330613s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347917313 (571h53m41.219000348s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.348510613 (571h53m41.219593648s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.503950985 (571h53m41.37503402s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.561352025 (571h53m41.43243506s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.622447380 (571h53m41.493530425s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.645494685 (571h53m41.51657773s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.247930045 (571h53m42.11901308s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.262221890 (571h53m42.133304925s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.295598123 (571h53m42.166681158s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.307481351 (571h53m42.178564386s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325775906 (571h53m42.196858941s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325865978 (571h53m42.196949023s) [seg_pos={0 0}, upload=501384]: Segment reached optimal threshold of 14 pieces. Dec 29 11:22:42.326137074 (571h53m42.197220109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection Dec 29 11:22:42.326153074 (571h53m42.197236109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326193401 (571h53m42.197276446s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326205705 (571h53m42.19728874s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326407739 (571h53m42.197490784s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326415374 (571h53m42.197498419s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326505105 (571h53m42.19758815s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326512188 (571h53m42.197595223s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326532357 (571h53m42.197615392s) [seg_pos={0 0}, upload=501384]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 11:22:42.326671562 (571h53m42.197754597s) [upload=501384]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:22:42.326677843 (571h53m42.197760879s) [upload=501384]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:22:42.610391438 (571h53m42.481474493s) [upload=501384]: Done uploading segments... Dec 29 11:22:42.610416826 (571h53m42.481499871s) [upload=501384]: Upload finished. err: <nil> Duration: 39.368784894s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [X..XXX..X.X.XX....X..X.....X.X...X.X.X.X.XX..XX..X..XXX.X...] duration: 5.669378028s trace: https://jaeger.dev.storj.io/trace/478381ee752feccc FAIL (2025-12-29 11:25:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: HNWAPBAX36BBXIDGUPQDTDSQA4PZ4TGB22EUWV3RYGUHZCEZOE2Q) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: 4KU7BI4YDC62OWW452QQBBL6FPPQ4XFV6622H3S7NHVH3RC4QUNA) context deadline exceeded ======LOGS====== Dec 29 11:22:32.632698611 (571h53m32.503781656s) [upload=501383]: Starting upload Dec 29 11:22:32.632771780 (571h53m32.503854815s) [upload=501383]: Done starting upload Dec 29 11:22:32.632851863 (571h53m32.503934898s) [upload=501383]: Uploading segments... Dec 29 11:22:32.632999464 (571h53m32.504082509s) [upload=501383]: Got next segment. Inline: false Dec 29 11:22:32.633036064 (571h53m32.504119099s) [upload=501383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 11:22:32.633044419 (571h53m32.504127464s) [upload=501383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 11:22:32.881455231 (571h53m32.752538266s) [seg_pos={0 0}, upload=501383]: Begin upload segment... Dec 29 11:22:32.881502301 (571h53m32.752585336s) [seg_pos={0 0}, upload=501383]: Done begin upload segment. Dec 29 11:22:32.881527148 (571h53m32.752610183s) [noise=true , address=145.79.227.36:4519 , piece_id=CYHA5B5AAYLQYGG7PKDZBL6KBNL7PWAXC3XJESNZJZNG6VFGAUDQ, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.881534973 (571h53m32.752618008s) [noise=true , address=45.91.116.9:4264 , piece_id=O23BR37PP6JQXJVHRI65MNPHKXXBKEMXGEEKVKSFS3NJQIQEWX2Q, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.881630104 (571h53m32.752713139s) [noise=true , address=145.79.227.38:4978 , piece_id=WYIS25RQHPKBFK2T2NKYDLXXSRLTODHVGQESJS2EAGK76E7ATO3Q, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.882288178 (571h53m32.753371213s) [noise=true , address=145.79.227.3:4149 , piece_id=PO7SJV27DUHYSB73OAZ3OKAJYF4TVF4DWHT2HZAIC2A2YQVCKXXQ, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.882621682 (571h53m32.753704717s) [noise=true , address=77.90.31.10:4517 , piece_id=3QAS444K45QVWONIQJAJVQLKQPKD62ER2NUOFUAMUO7LCF7WWPGQ, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.883291408 (571h53m32.754374443s) [noise=true , address=45.91.116.8:4045 , piece_id=HNWAPBAX36BBXIDGUPQDTDSQA4PZ4TGB22EUWV3RYGUHZCEZOE2Q, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.883400165 (571h53m32.75448323s) [noise=true , address=45.91.116.10:4084 , piece_id=4KU7BI4YDC62OWW452QQBBL6FPPQ4XFV6622H3S7NHVH3RC4QUNA, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.883704695 (571h53m32.75478774s) [noise=true , address=145.79.227.16:4910 , piece_id=MR4IES6CULVSVQHYOYSNHUL6RWDIXQHES2S43HPMFLCKR726MCIA, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.884302744 (571h53m32.755385779s) [noise=true , address=145.79.227.50:4374 , piece_id=PYOD755FKX2UEJ2KGDSO77M4I5MY2GUGZ6WPKC5P73QYRQXTOYLQ, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.884750867 (571h53m32.755833912s) [noise=true , address=145.79.227.28:4726 , piece_id=H5FAH27QEOC6PMAZSX7FVEIMGI6JKTQAOYKMBXSFXRYNQT25324A, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.885400453 (571h53m32.756483489s) [noise=true , address=145.79.227.51:4409 , piece_id=3K5GWDCXZD2EU2L4J7J57IWCSGXHEMPFVO4YXH4SUCBGAFXOY7GA, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.886481201 (571h53m32.757564246s) [noise=true , address=45.91.116.14:4190 , piece_id=4LEIJANQSVBZH3XCRNKZ6LBQBK3P46Q6DOFD2VKFVCCBEIL7ECEA, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.887235819 (571h53m32.758318854s) [noise=true , address=145.79.227.19:4079 , piece_id=JCRBTXMT3G6GJA5K4XLZW2Y6KSHX5JAXW5GJGH6IMMWL5F47MNRQ, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.887920132 (571h53m32.759003167s) [noise=true , address=45.91.116.5:4297 , piece_id=VMKLHK4Z34NW75AVI4ZA2OA7ONSTY4C2RIR6T4AGANRCGHBIP4GA, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.889101662 (571h53m32.760184707s) [noise=true , address=145.79.227.37:4565 , piece_id=BSQN6OBVAEXEBBPDVSYAOQBG7ZNJUEMWSRIPSSIUZKVXWAI5QNEQ, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.889343362 (571h53m32.760426427s) [noise=true , address=145.79.227.6:4258 , piece_id=U6EG23QFBI3OR6ECAMRXZXZUWZ36RC5VY54LFXGWBEQN6PWNYBZA, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:34.576948050 (571h53m34.448031096s) [upload=501383]: Next returned nil segment Dec 29 11:22:34.576961886 (571h53m34.448044921s) [upload=501383]: Waiting for error group managing segments... Dec 29 11:22:34.861780225 (571h53m34.73286327s) [noise=true , address=145.79.227.3:4149 , piece_id=PO7SJV27DUHYSB73OAZ3OKAJYF4TVF4DWHT2HZAIC2A2YQVCKXXQ, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:34.862453887 (571h53m34.733536922s) [noise=true , address=145.79.227.37:4565 , piece_id=BSQN6OBVAEXEBBPDVSYAOQBG7ZNJUEMWSRIPSSIUZKVXWAI5QNEQ, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:34.863190951 (571h53m34.734273986s) [noise=true , address=145.79.227.38:4978 , piece_id=WYIS25RQHPKBFK2T2NKYDLXXSRLTODHVGQESJS2EAGK76E7ATO3Q, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:34.866655959 (571h53m34.737738994s) [noise=true , address=145.79.227.16:4910 , piece_id=MR4IES6CULVSVQHYOYSNHUL6RWDIXQHES2S43HPMFLCKR726MCIA, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:34.870192272 (571h53m34.741275317s) [noise=true , address=145.79.227.36:4519 , piece_id=CYHA5B5AAYLQYGG7PKDZBL6KBNL7PWAXC3XJESNZJZNG6VFGAUDQ, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:34.870875714 (571h53m34.741958749s) [noise=true , address=145.79.227.50:4374 , piece_id=PYOD755FKX2UEJ2KGDSO77M4I5MY2GUGZ6WPKC5P73QYRQXTOYLQ, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:34.872583666 (571h53m34.743666701s) [noise=true , address=145.79.227.28:4726 , piece_id=H5FAH27QEOC6PMAZSX7FVEIMGI6JKTQAOYKMBXSFXRYNQT25324A, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:34.890730350 (571h53m34.761813395s) [noise=true , address=145.79.227.51:4409 , piece_id=3K5GWDCXZD2EU2L4J7J57IWCSGXHEMPFVO4YXH4SUCBGAFXOY7GA, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:34.891395006 (571h53m34.762478051s) [noise=true , address=145.79.227.19:4079 , piece_id=JCRBTXMT3G6GJA5K4XLZW2Y6KSHX5JAXW5GJGH6IMMWL5F47MNRQ, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:34.895657111 (571h53m34.766740146s) [noise=true , address=145.79.227.6:4258 , piece_id=U6EG23QFBI3OR6ECAMRXZXZUWZ36RC5VY54LFXGWBEQN6PWNYBZA, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:35.227622103 (571h53m35.098705139s) [noise=true , address=45.91.116.5:4297 , piece_id=VMKLHK4Z34NW75AVI4ZA2OA7ONSTY4C2RIR6T4AGANRCGHBIP4GA, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:35.261404501 (571h53m35.132487556s) [noise=true , address=45.91.116.8:4045 , piece_id=HNWAPBAX36BBXIDGUPQDTDSQA4PZ4TGB22EUWV3RYGUHZCEZOE2Q, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:35.280892309 (571h53m35.151975344s) [noise=true , address=45.91.116.10:4084 , piece_id=4KU7BI4YDC62OWW452QQBBL6FPPQ4XFV6622H3S7NHVH3RC4QUNA, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:35.280914941 (571h53m35.151997986s) [seg_pos={0 0}, upload=501383]: Segment reached optimal threshold of 13 pieces. Dec 29 11:22:35.281224681 (571h53m35.152307746s) [noise=true , address=77.90.31.10:4517 , piece_id=3QAS444K45QVWONIQJAJVQLKQPKD62ER2NUOFUAMUO7LCF7WWPGQ, seg_pos={0 0}, upload=501383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:35.281234840 (571h53m35.152317885s) [noise=true , address=77.90.31.10:4517 , piece_id=3QAS444K45QVWONIQJAJVQLKQPKD62ER2NUOFUAMUO7LCF7WWPGQ, seg_pos={0 0}, upload=501383]: Long tail cancellation due to optimal threshold Dec 29 11:22:35.281401778 (571h53m35.152484823s) [noise=true , address=45.91.116.14:4190 , piece_id=4LEIJANQSVBZH3XCRNKZ6LBQBK3P46Q6DOFD2VKFVCCBEIL7ECEA, seg_pos={0 0}, upload=501383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:35.281424642 (571h53m35.152507677s) [noise=true , address=45.91.116.14:4190 , piece_id=4LEIJANQSVBZH3XCRNKZ6LBQBK3P46Q6DOFD2VKFVCCBEIL7ECEA, seg_pos={0 0}, upload=501383]: Long tail cancellation due to optimal threshold Dec 29 11:22:35.281569658 (571h53m35.152652703s) [noise=true , address=45.91.116.9:4264 , piece_id=O23BR37PP6JQXJVHRI65MNPHKXXBKEMXGEEKVKSFS3NJQIQEWX2Q, seg_pos={0 0}, upload=501383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:35.281578355 (571h53m35.15266139s) [noise=true , address=45.91.116.9:4264 , piece_id=O23BR37PP6JQXJVHRI65MNPHKXXBKEMXGEEKVKSFS3NJQIQEWX2Q, seg_pos={0 0}, upload=501383]: Long tail cancellation due to optimal threshold Dec 29 11:22:35.281598103 (571h53m35.152681148s) [seg_pos={0 0}, upload=501383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 11:22:35.281676261 (571h53m35.152759316s) [upload=501383]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:22:35.281682853 (571h53m35.152765889s) [upload=501383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:22:35.572843998 (571h53m35.443927044s) [upload=501383]: Done uploading segments... Dec 29 11:22:35.572864308 (571h53m35.443947353s) [upload=501383]: Upload finished. err: <nil> Duration: 5.669478229s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X..................................X...............X..X...] duration: 15.672295882s trace: https://jaeger.dev.storj.io/trace/7e7354d8023608a8 FAIL (2025-12-29 11:25:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== ) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480097317 (571h53m39.351180352s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480405053 (571h53m39.351488098s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480712969 (571h53m39.351796014s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481019453 (571h53m39.352102488s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481029532 (571h53m39.352112567s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481559832 (571h53m39.352642877s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481731919 (571h53m39.352814954s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482078329 (571h53m39.353161364s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482099830 (571h53m39.353182865s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482434487 (571h53m39.353517532s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482480845 (571h53m39.35356388s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482841442 (571h53m39.353924477s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483163645 (571h53m39.35424668s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483532478 (571h53m39.354615513s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483914596 (571h53m39.354997631s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:40.761721106 (571h53m40.632804141s) [upload=501384]: Next returned nil segment Dec 29 11:22:40.761733349 (571h53m40.632816384s) [upload=501384]: Waiting for error group managing segments... Dec 29 11:22:41.121222862 (571h53m40.992305897s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.197311646 (571h53m41.068394691s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347247568 (571h53m41.218330613s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347917313 (571h53m41.219000348s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.348510613 (571h53m41.219593648s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.503950985 (571h53m41.37503402s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.561352025 (571h53m41.43243506s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.622447380 (571h53m41.493530425s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.645494685 (571h53m41.51657773s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.247930045 (571h53m42.11901308s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.262221890 (571h53m42.133304925s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.295598123 (571h53m42.166681158s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.307481351 (571h53m42.178564386s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325775906 (571h53m42.196858941s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325865978 (571h53m42.196949023s) [seg_pos={0 0}, upload=501384]: Segment reached optimal threshold of 14 pieces. Dec 29 11:22:42.326137074 (571h53m42.197220109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection Dec 29 11:22:42.326153074 (571h53m42.197236109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326193401 (571h53m42.197276446s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326205705 (571h53m42.19728874s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326407739 (571h53m42.197490784s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326415374 (571h53m42.197498419s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326505105 (571h53m42.19758815s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326512188 (571h53m42.197595223s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326532357 (571h53m42.197615392s) [seg_pos={0 0}, upload=501384]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 11:22:42.326671562 (571h53m42.197754597s) [upload=501384]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:22:42.326677843 (571h53m42.197760879s) [upload=501384]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:22:42.610391438 (571h53m42.481474493s) [upload=501384]: Done uploading segments... Dec 29 11:22:42.610416826 (571h53m42.481499871s) [upload=501384]: Upload finished. err: <nil> Duration: 15.672314327s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...X..................................X...............X..X..] duration: 38.098081408s trace: https://jaeger.dev.storj.io/trace/7e7354d8023608a8 FAIL (2025-12-29 11:25:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== ) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480097317 (571h53m39.351180352s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480405053 (571h53m39.351488098s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480712969 (571h53m39.351796014s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481019453 (571h53m39.352102488s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481029532 (571h53m39.352112567s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481559832 (571h53m39.352642877s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481731919 (571h53m39.352814954s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482078329 (571h53m39.353161364s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482099830 (571h53m39.353182865s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482434487 (571h53m39.353517532s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482480845 (571h53m39.35356388s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482841442 (571h53m39.353924477s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483163645 (571h53m39.35424668s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483532478 (571h53m39.354615513s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483914596 (571h53m39.354997631s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:40.761721106 (571h53m40.632804141s) [upload=501384]: Next returned nil segment Dec 29 11:22:40.761733349 (571h53m40.632816384s) [upload=501384]: Waiting for error group managing segments... Dec 29 11:22:41.121222862 (571h53m40.992305897s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.197311646 (571h53m41.068394691s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347247568 (571h53m41.218330613s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347917313 (571h53m41.219000348s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.348510613 (571h53m41.219593648s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.503950985 (571h53m41.37503402s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.561352025 (571h53m41.43243506s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.622447380 (571h53m41.493530425s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.645494685 (571h53m41.51657773s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.247930045 (571h53m42.11901308s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.262221890 (571h53m42.133304925s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.295598123 (571h53m42.166681158s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.307481351 (571h53m42.178564386s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325775906 (571h53m42.196858941s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325865978 (571h53m42.196949023s) [seg_pos={0 0}, upload=501384]: Segment reached optimal threshold of 14 pieces. Dec 29 11:22:42.326137074 (571h53m42.197220109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection Dec 29 11:22:42.326153074 (571h53m42.197236109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326193401 (571h53m42.197276446s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326205705 (571h53m42.19728874s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326407739 (571h53m42.197490784s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326415374 (571h53m42.197498419s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326505105 (571h53m42.19758815s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326512188 (571h53m42.197595223s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326532357 (571h53m42.197615392s) [seg_pos={0 0}, upload=501384]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 11:22:42.326671562 (571h53m42.197754597s) [upload=501384]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:22:42.326677843 (571h53m42.197760879s) [upload=501384]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:22:42.610391438 (571h53m42.481474493s) [upload=501384]: Done uploading segments... Dec 29 11:22:42.610416826 (571h53m42.481499871s) [upload=501384]: Upload finished. err: <nil> Duration: 38.098103481s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [....X...X....X...X..........X....X.......X..X...XX..XX......] duration: 3.923482815s trace: https://jaeger.dev.storj.io/trace/6c77d78dac3d14a1 FAIL (2025-12-29 11:25:23 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.923508713s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [.....X...X....X...X..........X....X.......X..X...XX..XX.....] duration: 4.894162197s trace: https://jaeger.dev.storj.io/trace/6c77d78dac3d14a1 FAIL (2025-12-29 11:25:06 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.894189608s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [.X..XXX..X.X.XX....X..X.....X.X...X.X.X.X.XX..XX..X..XXX.X..] duration: 5.918768843s trace: https://jaeger.dev.storj.io/trace/478381ee752feccc FAIL (2025-12-29 11:24:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: HNWAPBAX36BBXIDGUPQDTDSQA4PZ4TGB22EUWV3RYGUHZCEZOE2Q) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: 4KU7BI4YDC62OWW452QQBBL6FPPQ4XFV6622H3S7NHVH3RC4QUNA) context deadline exceeded ======LOGS====== Dec 29 11:22:32.632698611 (571h53m32.503781656s) [upload=501383]: Starting upload Dec 29 11:22:32.632771780 (571h53m32.503854815s) [upload=501383]: Done starting upload Dec 29 11:22:32.632851863 (571h53m32.503934898s) [upload=501383]: Uploading segments... Dec 29 11:22:32.632999464 (571h53m32.504082509s) [upload=501383]: Got next segment. Inline: false Dec 29 11:22:32.633036064 (571h53m32.504119099s) [upload=501383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 11:22:32.633044419 (571h53m32.504127464s) [upload=501383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 11:22:32.881455231 (571h53m32.752538266s) [seg_pos={0 0}, upload=501383]: Begin upload segment... Dec 29 11:22:32.881502301 (571h53m32.752585336s) [seg_pos={0 0}, upload=501383]: Done begin upload segment. Dec 29 11:22:32.881527148 (571h53m32.752610183s) [noise=true , address=145.79.227.36:4519 , piece_id=CYHA5B5AAYLQYGG7PKDZBL6KBNL7PWAXC3XJESNZJZNG6VFGAUDQ, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.881534973 (571h53m32.752618008s) [noise=true , address=45.91.116.9:4264 , piece_id=O23BR37PP6JQXJVHRI65MNPHKXXBKEMXGEEKVKSFS3NJQIQEWX2Q, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.881630104 (571h53m32.752713139s) [noise=true , address=145.79.227.38:4978 , piece_id=WYIS25RQHPKBFK2T2NKYDLXXSRLTODHVGQESJS2EAGK76E7ATO3Q, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.882288178 (571h53m32.753371213s) [noise=true , address=145.79.227.3:4149 , piece_id=PO7SJV27DUHYSB73OAZ3OKAJYF4TVF4DWHT2HZAIC2A2YQVCKXXQ, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.882621682 (571h53m32.753704717s) [noise=true , address=77.90.31.10:4517 , piece_id=3QAS444K45QVWONIQJAJVQLKQPKD62ER2NUOFUAMUO7LCF7WWPGQ, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.883291408 (571h53m32.754374443s) [noise=true , address=45.91.116.8:4045 , piece_id=HNWAPBAX36BBXIDGUPQDTDSQA4PZ4TGB22EUWV3RYGUHZCEZOE2Q, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.883400165 (571h53m32.75448323s) [noise=true , address=45.91.116.10:4084 , piece_id=4KU7BI4YDC62OWW452QQBBL6FPPQ4XFV6622H3S7NHVH3RC4QUNA, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.883704695 (571h53m32.75478774s) [noise=true , address=145.79.227.16:4910 , piece_id=MR4IES6CULVSVQHYOYSNHUL6RWDIXQHES2S43HPMFLCKR726MCIA, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.884302744 (571h53m32.755385779s) [noise=true , address=145.79.227.50:4374 , piece_id=PYOD755FKX2UEJ2KGDSO77M4I5MY2GUGZ6WPKC5P73QYRQXTOYLQ, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.884750867 (571h53m32.755833912s) [noise=true , address=145.79.227.28:4726 , piece_id=H5FAH27QEOC6PMAZSX7FVEIMGI6JKTQAOYKMBXSFXRYNQT25324A, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.885400453 (571h53m32.756483489s) [noise=true , address=145.79.227.51:4409 , piece_id=3K5GWDCXZD2EU2L4J7J57IWCSGXHEMPFVO4YXH4SUCBGAFXOY7GA, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.886481201 (571h53m32.757564246s) [noise=true , address=45.91.116.14:4190 , piece_id=4LEIJANQSVBZH3XCRNKZ6LBQBK3P46Q6DOFD2VKFVCCBEIL7ECEA, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.887235819 (571h53m32.758318854s) [noise=true , address=145.79.227.19:4079 , piece_id=JCRBTXMT3G6GJA5K4XLZW2Y6KSHX5JAXW5GJGH6IMMWL5F47MNRQ, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.887920132 (571h53m32.759003167s) [noise=true , address=45.91.116.5:4297 , piece_id=VMKLHK4Z34NW75AVI4ZA2OA7ONSTY4C2RIR6T4AGANRCGHBIP4GA, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.889101662 (571h53m32.760184707s) [noise=true , address=145.79.227.37:4565 , piece_id=BSQN6OBVAEXEBBPDVSYAOQBG7ZNJUEMWSRIPSSIUZKVXWAI5QNEQ, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.889343362 (571h53m32.760426427s) [noise=true , address=145.79.227.6:4258 , piece_id=U6EG23QFBI3OR6ECAMRXZXZUWZ36RC5VY54LFXGWBEQN6PWNYBZA, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:34.576948050 (571h53m34.448031096s) [upload=501383]: Next returned nil segment Dec 29 11:22:34.576961886 (571h53m34.448044921s) [upload=501383]: Waiting for error group managing segments... Dec 29 11:22:34.861780225 (571h53m34.73286327s) [noise=true , address=145.79.227.3:4149 , piece_id=PO7SJV27DUHYSB73OAZ3OKAJYF4TVF4DWHT2HZAIC2A2YQVCKXXQ, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:34.862453887 (571h53m34.733536922s) [noise=true , address=145.79.227.37:4565 , piece_id=BSQN6OBVAEXEBBPDVSYAOQBG7ZNJUEMWSRIPSSIUZKVXWAI5QNEQ, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:34.863190951 (571h53m34.734273986s) [noise=true , address=145.79.227.38:4978 , piece_id=WYIS25RQHPKBFK2T2NKYDLXXSRLTODHVGQESJS2EAGK76E7ATO3Q, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:34.866655959 (571h53m34.737738994s) [noise=true , address=145.79.227.16:4910 , piece_id=MR4IES6CULVSVQHYOYSNHUL6RWDIXQHES2S43HPMFLCKR726MCIA, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:34.870192272 (571h53m34.741275317s) [noise=true , address=145.79.227.36:4519 , piece_id=CYHA5B5AAYLQYGG7PKDZBL6KBNL7PWAXC3XJESNZJZNG6VFGAUDQ, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:34.870875714 (571h53m34.741958749s) [noise=true , address=145.79.227.50:4374 , piece_id=PYOD755FKX2UEJ2KGDSO77M4I5MY2GUGZ6WPKC5P73QYRQXTOYLQ, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:34.872583666 (571h53m34.743666701s) [noise=true , address=145.79.227.28:4726 , piece_id=H5FAH27QEOC6PMAZSX7FVEIMGI6JKTQAOYKMBXSFXRYNQT25324A, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:34.890730350 (571h53m34.761813395s) [noise=true , address=145.79.227.51:4409 , piece_id=3K5GWDCXZD2EU2L4J7J57IWCSGXHEMPFVO4YXH4SUCBGAFXOY7GA, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:34.891395006 (571h53m34.762478051s) [noise=true , address=145.79.227.19:4079 , piece_id=JCRBTXMT3G6GJA5K4XLZW2Y6KSHX5JAXW5GJGH6IMMWL5F47MNRQ, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:34.895657111 (571h53m34.766740146s) [noise=true , address=145.79.227.6:4258 , piece_id=U6EG23QFBI3OR6ECAMRXZXZUWZ36RC5VY54LFXGWBEQN6PWNYBZA, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:35.227622103 (571h53m35.098705139s) [noise=true , address=45.91.116.5:4297 , piece_id=VMKLHK4Z34NW75AVI4ZA2OA7ONSTY4C2RIR6T4AGANRCGHBIP4GA, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:35.261404501 (571h53m35.132487556s) [noise=true , address=45.91.116.8:4045 , piece_id=HNWAPBAX36BBXIDGUPQDTDSQA4PZ4TGB22EUWV3RYGUHZCEZOE2Q, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:35.280892309 (571h53m35.151975344s) [noise=true , address=45.91.116.10:4084 , piece_id=4KU7BI4YDC62OWW452QQBBL6FPPQ4XFV6622H3S7NHVH3RC4QUNA, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:35.280914941 (571h53m35.151997986s) [seg_pos={0 0}, upload=501383]: Segment reached optimal threshold of 13 pieces. Dec 29 11:22:35.281224681 (571h53m35.152307746s) [noise=true , address=77.90.31.10:4517 , piece_id=3QAS444K45QVWONIQJAJVQLKQPKD62ER2NUOFUAMUO7LCF7WWPGQ, seg_pos={0 0}, upload=501383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:35.281234840 (571h53m35.152317885s) [noise=true , address=77.90.31.10:4517 , piece_id=3QAS444K45QVWONIQJAJVQLKQPKD62ER2NUOFUAMUO7LCF7WWPGQ, seg_pos={0 0}, upload=501383]: Long tail cancellation due to optimal threshold Dec 29 11:22:35.281401778 (571h53m35.152484823s) [noise=true , address=45.91.116.14:4190 , piece_id=4LEIJANQSVBZH3XCRNKZ6LBQBK3P46Q6DOFD2VKFVCCBEIL7ECEA, seg_pos={0 0}, upload=501383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:35.281424642 (571h53m35.152507677s) [noise=true , address=45.91.116.14:4190 , piece_id=4LEIJANQSVBZH3XCRNKZ6LBQBK3P46Q6DOFD2VKFVCCBEIL7ECEA, seg_pos={0 0}, upload=501383]: Long tail cancellation due to optimal threshold Dec 29 11:22:35.281569658 (571h53m35.152652703s) [noise=true , address=45.91.116.9:4264 , piece_id=O23BR37PP6JQXJVHRI65MNPHKXXBKEMXGEEKVKSFS3NJQIQEWX2Q, seg_pos={0 0}, upload=501383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:35.281578355 (571h53m35.15266139s) [noise=true , address=45.91.116.9:4264 , piece_id=O23BR37PP6JQXJVHRI65MNPHKXXBKEMXGEEKVKSFS3NJQIQEWX2Q, seg_pos={0 0}, upload=501383]: Long tail cancellation due to optimal threshold Dec 29 11:22:35.281598103 (571h53m35.152681148s) [seg_pos={0 0}, upload=501383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 11:22:35.281676261 (571h53m35.152759316s) [upload=501383]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:22:35.281682853 (571h53m35.152765889s) [upload=501383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:22:35.572843998 (571h53m35.443927044s) [upload=501383]: Done uploading segments... Dec 29 11:22:35.572864308 (571h53m35.443947353s) [upload=501383]: Upload finished. err: <nil> Duration: 5.918799832s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [....X..................................X...............X..X.] duration: 38.844556848s trace: https://jaeger.dev.storj.io/trace/7e7354d8023608a8 FAIL (2025-12-29 11:24:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== ) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480097317 (571h53m39.351180352s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480405053 (571h53m39.351488098s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480712969 (571h53m39.351796014s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481019453 (571h53m39.352102488s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481029532 (571h53m39.352112567s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481559832 (571h53m39.352642877s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481731919 (571h53m39.352814954s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482078329 (571h53m39.353161364s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482099830 (571h53m39.353182865s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482434487 (571h53m39.353517532s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482480845 (571h53m39.35356388s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482841442 (571h53m39.353924477s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483163645 (571h53m39.35424668s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483532478 (571h53m39.354615513s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483914596 (571h53m39.354997631s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:40.761721106 (571h53m40.632804141s) [upload=501384]: Next returned nil segment Dec 29 11:22:40.761733349 (571h53m40.632816384s) [upload=501384]: Waiting for error group managing segments... Dec 29 11:22:41.121222862 (571h53m40.992305897s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.197311646 (571h53m41.068394691s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347247568 (571h53m41.218330613s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347917313 (571h53m41.219000348s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.348510613 (571h53m41.219593648s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.503950985 (571h53m41.37503402s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.561352025 (571h53m41.43243506s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.622447380 (571h53m41.493530425s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.645494685 (571h53m41.51657773s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.247930045 (571h53m42.11901308s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.262221890 (571h53m42.133304925s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.295598123 (571h53m42.166681158s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.307481351 (571h53m42.178564386s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325775906 (571h53m42.196858941s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325865978 (571h53m42.196949023s) [seg_pos={0 0}, upload=501384]: Segment reached optimal threshold of 14 pieces. Dec 29 11:22:42.326137074 (571h53m42.197220109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection Dec 29 11:22:42.326153074 (571h53m42.197236109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326193401 (571h53m42.197276446s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326205705 (571h53m42.19728874s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326407739 (571h53m42.197490784s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326415374 (571h53m42.197498419s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326505105 (571h53m42.19758815s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326512188 (571h53m42.197595223s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326532357 (571h53m42.197615392s) [seg_pos={0 0}, upload=501384]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 11:22:42.326671562 (571h53m42.197754597s) [upload=501384]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:22:42.326677843 (571h53m42.197760879s) [upload=501384]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:22:42.610391438 (571h53m42.481474493s) [upload=501384]: Done uploading segments... Dec 29 11:22:42.610416826 (571h53m42.481499871s) [upload=501384]: Upload finished. err: <nil> Duration: 38.844574231s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [......X...X....X...X..........X....X.......X..X...XX..XX....] duration: 3.525965442s trace: https://jaeger.dev.storj.io/trace/6c77d78dac3d14a1 FAIL (2025-12-29 11:23:59 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.526004024s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [X.X..XXX..X.X.XX....X..X.....X.X...X.X.X.X.XX..XX..X..XXX.X.] duration: 5.982533906s trace: https://jaeger.dev.storj.io/trace/478381ee752feccc FAIL (2025-12-29 11:23:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: HNWAPBAX36BBXIDGUPQDTDSQA4PZ4TGB22EUWV3RYGUHZCEZOE2Q) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: 4KU7BI4YDC62OWW452QQBBL6FPPQ4XFV6622H3S7NHVH3RC4QUNA) context deadline exceeded ======LOGS====== Dec 29 11:22:32.632698611 (571h53m32.503781656s) [upload=501383]: Starting upload Dec 29 11:22:32.632771780 (571h53m32.503854815s) [upload=501383]: Done starting upload Dec 29 11:22:32.632851863 (571h53m32.503934898s) [upload=501383]: Uploading segments... Dec 29 11:22:32.632999464 (571h53m32.504082509s) [upload=501383]: Got next segment. Inline: false Dec 29 11:22:32.633036064 (571h53m32.504119099s) [upload=501383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 11:22:32.633044419 (571h53m32.504127464s) [upload=501383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 11:22:32.881455231 (571h53m32.752538266s) [seg_pos={0 0}, upload=501383]: Begin upload segment... Dec 29 11:22:32.881502301 (571h53m32.752585336s) [seg_pos={0 0}, upload=501383]: Done begin upload segment. Dec 29 11:22:32.881527148 (571h53m32.752610183s) [noise=true , address=145.79.227.36:4519 , piece_id=CYHA5B5AAYLQYGG7PKDZBL6KBNL7PWAXC3XJESNZJZNG6VFGAUDQ, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.881534973 (571h53m32.752618008s) [noise=true , address=45.91.116.9:4264 , piece_id=O23BR37PP6JQXJVHRI65MNPHKXXBKEMXGEEKVKSFS3NJQIQEWX2Q, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.881630104 (571h53m32.752713139s) [noise=true , address=145.79.227.38:4978 , piece_id=WYIS25RQHPKBFK2T2NKYDLXXSRLTODHVGQESJS2EAGK76E7ATO3Q, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.882288178 (571h53m32.753371213s) [noise=true , address=145.79.227.3:4149 , piece_id=PO7SJV27DUHYSB73OAZ3OKAJYF4TVF4DWHT2HZAIC2A2YQVCKXXQ, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.882621682 (571h53m32.753704717s) [noise=true , address=77.90.31.10:4517 , piece_id=3QAS444K45QVWONIQJAJVQLKQPKD62ER2NUOFUAMUO7LCF7WWPGQ, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.883291408 (571h53m32.754374443s) [noise=true , address=45.91.116.8:4045 , piece_id=HNWAPBAX36BBXIDGUPQDTDSQA4PZ4TGB22EUWV3RYGUHZCEZOE2Q, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.883400165 (571h53m32.75448323s) [noise=true , address=45.91.116.10:4084 , piece_id=4KU7BI4YDC62OWW452QQBBL6FPPQ4XFV6622H3S7NHVH3RC4QUNA, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.883704695 (571h53m32.75478774s) [noise=true , address=145.79.227.16:4910 , piece_id=MR4IES6CULVSVQHYOYSNHUL6RWDIXQHES2S43HPMFLCKR726MCIA, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.884302744 (571h53m32.755385779s) [noise=true , address=145.79.227.50:4374 , piece_id=PYOD755FKX2UEJ2KGDSO77M4I5MY2GUGZ6WPKC5P73QYRQXTOYLQ, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.884750867 (571h53m32.755833912s) [noise=true , address=145.79.227.28:4726 , piece_id=H5FAH27QEOC6PMAZSX7FVEIMGI6JKTQAOYKMBXSFXRYNQT25324A, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.885400453 (571h53m32.756483489s) [noise=true , address=145.79.227.51:4409 , piece_id=3K5GWDCXZD2EU2L4J7J57IWCSGXHEMPFVO4YXH4SUCBGAFXOY7GA, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.886481201 (571h53m32.757564246s) [noise=true , address=45.91.116.14:4190 , piece_id=4LEIJANQSVBZH3XCRNKZ6LBQBK3P46Q6DOFD2VKFVCCBEIL7ECEA, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.887235819 (571h53m32.758318854s) [noise=true , address=145.79.227.19:4079 , piece_id=JCRBTXMT3G6GJA5K4XLZW2Y6KSHX5JAXW5GJGH6IMMWL5F47MNRQ, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.887920132 (571h53m32.759003167s) [noise=true , address=45.91.116.5:4297 , piece_id=VMKLHK4Z34NW75AVI4ZA2OA7ONSTY4C2RIR6T4AGANRCGHBIP4GA, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.889101662 (571h53m32.760184707s) [noise=true , address=145.79.227.37:4565 , piece_id=BSQN6OBVAEXEBBPDVSYAOQBG7ZNJUEMWSRIPSSIUZKVXWAI5QNEQ, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.889343362 (571h53m32.760426427s) [noise=true , address=145.79.227.6:4258 , piece_id=U6EG23QFBI3OR6ECAMRXZXZUWZ36RC5VY54LFXGWBEQN6PWNYBZA, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:34.576948050 (571h53m34.448031096s) [upload=501383]: Next returned nil segment Dec 29 11:22:34.576961886 (571h53m34.448044921s) [upload=501383]: Waiting for error group managing segments... Dec 29 11:22:34.861780225 (571h53m34.73286327s) [noise=true , address=145.79.227.3:4149 , piece_id=PO7SJV27DUHYSB73OAZ3OKAJYF4TVF4DWHT2HZAIC2A2YQVCKXXQ, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:34.862453887 (571h53m34.733536922s) [noise=true , address=145.79.227.37:4565 , piece_id=BSQN6OBVAEXEBBPDVSYAOQBG7ZNJUEMWSRIPSSIUZKVXWAI5QNEQ, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:34.863190951 (571h53m34.734273986s) [noise=true , address=145.79.227.38:4978 , piece_id=WYIS25RQHPKBFK2T2NKYDLXXSRLTODHVGQESJS2EAGK76E7ATO3Q, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:34.866655959 (571h53m34.737738994s) [noise=true , address=145.79.227.16:4910 , piece_id=MR4IES6CULVSVQHYOYSNHUL6RWDIXQHES2S43HPMFLCKR726MCIA, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:34.870192272 (571h53m34.741275317s) [noise=true , address=145.79.227.36:4519 , piece_id=CYHA5B5AAYLQYGG7PKDZBL6KBNL7PWAXC3XJESNZJZNG6VFGAUDQ, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:34.870875714 (571h53m34.741958749s) [noise=true , address=145.79.227.50:4374 , piece_id=PYOD755FKX2UEJ2KGDSO77M4I5MY2GUGZ6WPKC5P73QYRQXTOYLQ, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:34.872583666 (571h53m34.743666701s) [noise=true , address=145.79.227.28:4726 , piece_id=H5FAH27QEOC6PMAZSX7FVEIMGI6JKTQAOYKMBXSFXRYNQT25324A, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:34.890730350 (571h53m34.761813395s) [noise=true , address=145.79.227.51:4409 , piece_id=3K5GWDCXZD2EU2L4J7J57IWCSGXHEMPFVO4YXH4SUCBGAFXOY7GA, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:34.891395006 (571h53m34.762478051s) [noise=true , address=145.79.227.19:4079 , piece_id=JCRBTXMT3G6GJA5K4XLZW2Y6KSHX5JAXW5GJGH6IMMWL5F47MNRQ, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:34.895657111 (571h53m34.766740146s) [noise=true , address=145.79.227.6:4258 , piece_id=U6EG23QFBI3OR6ECAMRXZXZUWZ36RC5VY54LFXGWBEQN6PWNYBZA, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:35.227622103 (571h53m35.098705139s) [noise=true , address=45.91.116.5:4297 , piece_id=VMKLHK4Z34NW75AVI4ZA2OA7ONSTY4C2RIR6T4AGANRCGHBIP4GA, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:35.261404501 (571h53m35.132487556s) [noise=true , address=45.91.116.8:4045 , piece_id=HNWAPBAX36BBXIDGUPQDTDSQA4PZ4TGB22EUWV3RYGUHZCEZOE2Q, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:35.280892309 (571h53m35.151975344s) [noise=true , address=45.91.116.10:4084 , piece_id=4KU7BI4YDC62OWW452QQBBL6FPPQ4XFV6622H3S7NHVH3RC4QUNA, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:35.280914941 (571h53m35.151997986s) [seg_pos={0 0}, upload=501383]: Segment reached optimal threshold of 13 pieces. Dec 29 11:22:35.281224681 (571h53m35.152307746s) [noise=true , address=77.90.31.10:4517 , piece_id=3QAS444K45QVWONIQJAJVQLKQPKD62ER2NUOFUAMUO7LCF7WWPGQ, seg_pos={0 0}, upload=501383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:35.281234840 (571h53m35.152317885s) [noise=true , address=77.90.31.10:4517 , piece_id=3QAS444K45QVWONIQJAJVQLKQPKD62ER2NUOFUAMUO7LCF7WWPGQ, seg_pos={0 0}, upload=501383]: Long tail cancellation due to optimal threshold Dec 29 11:22:35.281401778 (571h53m35.152484823s) [noise=true , address=45.91.116.14:4190 , piece_id=4LEIJANQSVBZH3XCRNKZ6LBQBK3P46Q6DOFD2VKFVCCBEIL7ECEA, seg_pos={0 0}, upload=501383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:35.281424642 (571h53m35.152507677s) [noise=true , address=45.91.116.14:4190 , piece_id=4LEIJANQSVBZH3XCRNKZ6LBQBK3P46Q6DOFD2VKFVCCBEIL7ECEA, seg_pos={0 0}, upload=501383]: Long tail cancellation due to optimal threshold Dec 29 11:22:35.281569658 (571h53m35.152652703s) [noise=true , address=45.91.116.9:4264 , piece_id=O23BR37PP6JQXJVHRI65MNPHKXXBKEMXGEEKVKSFS3NJQIQEWX2Q, seg_pos={0 0}, upload=501383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:35.281578355 (571h53m35.15266139s) [noise=true , address=45.91.116.9:4264 , piece_id=O23BR37PP6JQXJVHRI65MNPHKXXBKEMXGEEKVKSFS3NJQIQEWX2Q, seg_pos={0 0}, upload=501383]: Long tail cancellation due to optimal threshold Dec 29 11:22:35.281598103 (571h53m35.152681148s) [seg_pos={0 0}, upload=501383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 11:22:35.281676261 (571h53m35.152759316s) [upload=501383]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:22:35.281682853 (571h53m35.152765889s) [upload=501383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:22:35.572843998 (571h53m35.443927044s) [upload=501383]: Done uploading segments... Dec 29 11:22:35.572864308 (571h53m35.443947353s) [upload=501383]: Upload finished. err: <nil> Duration: 5.982558062s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....X..................................X...............X..X] duration: 59.302296391s trace: https://jaeger.dev.storj.io/trace/7e7354d8023608a8 FAIL (2025-12-29 11:23:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== ) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480097317 (571h53m39.351180352s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480405053 (571h53m39.351488098s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.480712969 (571h53m39.351796014s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481019453 (571h53m39.352102488s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481029532 (571h53m39.352112567s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481559832 (571h53m39.352642877s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.481731919 (571h53m39.352814954s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482078329 (571h53m39.353161364s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482099830 (571h53m39.353182865s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482434487 (571h53m39.353517532s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482480845 (571h53m39.35356388s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.482841442 (571h53m39.353924477s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483163645 (571h53m39.35424668s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483532478 (571h53m39.354615513s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:39.483914596 (571h53m39.354997631s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Uploading piece... Dec 29 11:22:40.761721106 (571h53m40.632804141s) [upload=501384]: Next returned nil segment Dec 29 11:22:40.761733349 (571h53m40.632816384s) [upload=501384]: Waiting for error group managing segments... Dec 29 11:22:41.121222862 (571h53m40.992305897s) [noise=true , address=64.49.13.140:4516 , piece_id=4GUYOYOAEASUK2W25S3NILWIIIHGOZX76L4EGFLUUUEDWRALPP2A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.197311646 (571h53m41.068394691s) [noise=true , address=64.49.13.26:4953 , piece_id=4YSW5Y3A6HYG3XJ2XAB7RGVHAOHMP7G3QIZYMYTSXVXYJ3ANLNXA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347247568 (571h53m41.218330613s) [noise=true , address=145.79.227.13:4807 , piece_id=VKXB5MWAG2YYH2AWWJEGTLBZ2QUPLI25VTHI6AMMALSM23Z3AWZQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.347917313 (571h53m41.219000348s) [noise=true , address=145.79.227.3:4164 , piece_id=U6F7VDPOH7K6R5EH5UUXHFP55GXOEXO5BLBJHZRTITPDQI4LP7KQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.348510613 (571h53m41.219593648s) [noise=true , address=145.79.227.29:4974 , piece_id=KEKREOD2B35XQNI45JBHE2SZZC3MQQBQUR5I6WFDAQYJQGJBXSMA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.503950985 (571h53m41.37503402s) [noise=true , address=77.90.31.17:4769 , piece_id=3MKUMOQIUVERKQLLIFX52XQPTQ5LSFMHIPD5YW2YTO5YEFR36IYA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.561352025 (571h53m41.43243506s) [noise=true , address=77.90.31.18:4834 , piece_id=BTWNGNQVEUEVV6HLBDM63SGNTKV4RDU4J4XFLG52WHKP53H3L5KA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.622447380 (571h53m41.493530425s) [noise=true , address=77.90.31.16:4750 , piece_id=MVVZPJEGDCBSNVOB6CD2WUZGIJTL3VBHX2G4L3TVZ6JF5ZEJ5U6Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:41.645494685 (571h53m41.51657773s) [noise=true , address=77.90.31.21:4889 , piece_id=BR6Z55P3A7YJ5FJRDSZFUIF66FGIJQY7WCEQ7PL5DOYWL73OMKTQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.247930045 (571h53m42.11901308s) [noise=true , address=145.79.227.30:4777 , piece_id=VSCXLXGIBSFE6E4OHLWNZIZ75PO7WQO2VFHUYHW7JUNH62KGGU7Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.262221890 (571h53m42.133304925s) [noise=true , address=145.79.227.44:4205 , piece_id=UE4QSHDQBZ3XBWQ6I6MCRVTLMU7U6W6OZDA2ZGQCH6ZYRAGPD7WA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.295598123 (571h53m42.166681158s) [noise=true , address=145.79.227.35:4515 , piece_id=L7LFZKZEZERUNX4HUNDNQ2PWHXNJRJ7OBFRNMFIICPZEUIBHGP7A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.307481351 (571h53m42.178564386s) [noise=true , address=145.79.227.34:4910 , piece_id=ZH4DCNXET2Q6VRAUH3NQ3KKSV4OYI575X7YS72XD236EEOMOG3LA, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325775906 (571h53m42.196858941s) [noise=true , address=145.79.227.40:4067 , piece_id=WSYDRX5YJ6OCFW52ZVMM6WZYNRFUXMR4XJ6C5727BOFMR4RRH33Q, seg_pos={0 0}, upload=501384]: Done uploading piece. err: <nil> Dec 29 11:22:42.325865978 (571h53m42.196949023s) [seg_pos={0 0}, upload=501384]: Segment reached optimal threshold of 14 pieces. Dec 29 11:22:42.326137074 (571h53m42.197220109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42560->64.49.13.169:4708: use of closed network connection Dec 29 11:22:42.326153074 (571h53m42.197236109s) [noise=true , address=64.49.13.169:4708 , piece_id=B2X2RWRELXWJPF6AVJJZCNU46LM5ASGRE4EWGPKL3SCLQ4VKUBDQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326193401 (571h53m42.197276446s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326205705 (571h53m42.19728874s) [noise=true , address=45.91.116.4:4969 , piece_id=BTZWZWE4HJT5XTA3CH3ZVBLDUTZ6N2KQ5VCSCZIU6FI4U3FIEPLQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326407739 (571h53m42.197490784s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326415374 (571h53m42.197498419s) [noise=true , address=77.90.31.10:4524 , piece_id=Q2JGEUV5EDFTVY2CSNPZOVJIIIAFBA3IHA7S5N7YZRH3E7TSTE5A, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326505105 (571h53m42.19758815s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:42.326512188 (571h53m42.197595223s) [noise=true , address=193.149.164.143:4700 , piece_id=Z7PSNZ5B22I55LAO7JXKGHDKTQGVJKZ4SQWVHJEY5GTQMOXO7NQQ, seg_pos={0 0}, upload=501384]: Long tail cancellation due to optimal threshold Dec 29 11:22:42.326532357 (571h53m42.197615392s) [seg_pos={0 0}, upload=501384]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 11:22:42.326671562 (571h53m42.197754597s) [upload=501384]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:22:42.326677843 (571h53m42.197760879s) [upload=501384]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:22:42.610391438 (571h53m42.481474493s) [upload=501384]: Done uploading segments... Dec 29 11:22:42.610416826 (571h53m42.481499871s) [upload=501384]: Upload finished. err: <nil> Duration: 59.302312132s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [X......X...X....X...X..........X....X.......X..X...XX..XX...] duration: 3.559867155s trace: https://jaeger.dev.storj.io/trace/6c77d78dac3d14a1 FAIL (2025-12-29 11:23:36 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.559890578s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [XX.X..XXX..X.X.XX....X..X.....X.X...X.X.X.X.XX..XX..X..XXX.X] duration: 30.002783794s trace: https://jaeger.dev.storj.io/trace/478381ee752feccc FAIL (2025-12-29 11:23:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: HNWAPBAX36BBXIDGUPQDTDSQA4PZ4TGB22EUWV3RYGUHZCEZOE2Q) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: 4KU7BI4YDC62OWW452QQBBL6FPPQ4XFV6622H3S7NHVH3RC4QUNA) context deadline exceeded ======LOGS====== Dec 29 11:22:32.632698611 (571h53m32.503781656s) [upload=501383]: Starting upload Dec 29 11:22:32.632771780 (571h53m32.503854815s) [upload=501383]: Done starting upload Dec 29 11:22:32.632851863 (571h53m32.503934898s) [upload=501383]: Uploading segments... Dec 29 11:22:32.632999464 (571h53m32.504082509s) [upload=501383]: Got next segment. Inline: false Dec 29 11:22:32.633036064 (571h53m32.504119099s) [upload=501383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 11:22:32.633044419 (571h53m32.504127464s) [upload=501383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 11:22:32.881455231 (571h53m32.752538266s) [seg_pos={0 0}, upload=501383]: Begin upload segment... Dec 29 11:22:32.881502301 (571h53m32.752585336s) [seg_pos={0 0}, upload=501383]: Done begin upload segment. Dec 29 11:22:32.881527148 (571h53m32.752610183s) [noise=true , address=145.79.227.36:4519 , piece_id=CYHA5B5AAYLQYGG7PKDZBL6KBNL7PWAXC3XJESNZJZNG6VFGAUDQ, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.881534973 (571h53m32.752618008s) [noise=true , address=45.91.116.9:4264 , piece_id=O23BR37PP6JQXJVHRI65MNPHKXXBKEMXGEEKVKSFS3NJQIQEWX2Q, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.881630104 (571h53m32.752713139s) [noise=true , address=145.79.227.38:4978 , piece_id=WYIS25RQHPKBFK2T2NKYDLXXSRLTODHVGQESJS2EAGK76E7ATO3Q, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.882288178 (571h53m32.753371213s) [noise=true , address=145.79.227.3:4149 , piece_id=PO7SJV27DUHYSB73OAZ3OKAJYF4TVF4DWHT2HZAIC2A2YQVCKXXQ, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.882621682 (571h53m32.753704717s) [noise=true , address=77.90.31.10:4517 , piece_id=3QAS444K45QVWONIQJAJVQLKQPKD62ER2NUOFUAMUO7LCF7WWPGQ, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.883291408 (571h53m32.754374443s) [noise=true , address=45.91.116.8:4045 , piece_id=HNWAPBAX36BBXIDGUPQDTDSQA4PZ4TGB22EUWV3RYGUHZCEZOE2Q, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.883400165 (571h53m32.75448323s) [noise=true , address=45.91.116.10:4084 , piece_id=4KU7BI4YDC62OWW452QQBBL6FPPQ4XFV6622H3S7NHVH3RC4QUNA, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.883704695 (571h53m32.75478774s) [noise=true , address=145.79.227.16:4910 , piece_id=MR4IES6CULVSVQHYOYSNHUL6RWDIXQHES2S43HPMFLCKR726MCIA, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.884302744 (571h53m32.755385779s) [noise=true , address=145.79.227.50:4374 , piece_id=PYOD755FKX2UEJ2KGDSO77M4I5MY2GUGZ6WPKC5P73QYRQXTOYLQ, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.884750867 (571h53m32.755833912s) [noise=true , address=145.79.227.28:4726 , piece_id=H5FAH27QEOC6PMAZSX7FVEIMGI6JKTQAOYKMBXSFXRYNQT25324A, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.885400453 (571h53m32.756483489s) [noise=true , address=145.79.227.51:4409 , piece_id=3K5GWDCXZD2EU2L4J7J57IWCSGXHEMPFVO4YXH4SUCBGAFXOY7GA, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.886481201 (571h53m32.757564246s) [noise=true , address=45.91.116.14:4190 , piece_id=4LEIJANQSVBZH3XCRNKZ6LBQBK3P46Q6DOFD2VKFVCCBEIL7ECEA, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.887235819 (571h53m32.758318854s) [noise=true , address=145.79.227.19:4079 , piece_id=JCRBTXMT3G6GJA5K4XLZW2Y6KSHX5JAXW5GJGH6IMMWL5F47MNRQ, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.887920132 (571h53m32.759003167s) [noise=true , address=45.91.116.5:4297 , piece_id=VMKLHK4Z34NW75AVI4ZA2OA7ONSTY4C2RIR6T4AGANRCGHBIP4GA, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.889101662 (571h53m32.760184707s) [noise=true , address=145.79.227.37:4565 , piece_id=BSQN6OBVAEXEBBPDVSYAOQBG7ZNJUEMWSRIPSSIUZKVXWAI5QNEQ, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:32.889343362 (571h53m32.760426427s) [noise=true , address=145.79.227.6:4258 , piece_id=U6EG23QFBI3OR6ECAMRXZXZUWZ36RC5VY54LFXGWBEQN6PWNYBZA, seg_pos={0 0}, upload=501383]: Uploading piece... Dec 29 11:22:34.576948050 (571h53m34.448031096s) [upload=501383]: Next returned nil segment Dec 29 11:22:34.576961886 (571h53m34.448044921s) [upload=501383]: Waiting for error group managing segments... Dec 29 11:22:34.861780225 (571h53m34.73286327s) [noise=true , address=145.79.227.3:4149 , piece_id=PO7SJV27DUHYSB73OAZ3OKAJYF4TVF4DWHT2HZAIC2A2YQVCKXXQ, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:34.862453887 (571h53m34.733536922s) [noise=true , address=145.79.227.37:4565 , piece_id=BSQN6OBVAEXEBBPDVSYAOQBG7ZNJUEMWSRIPSSIUZKVXWAI5QNEQ, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:34.863190951 (571h53m34.734273986s) [noise=true , address=145.79.227.38:4978 , piece_id=WYIS25RQHPKBFK2T2NKYDLXXSRLTODHVGQESJS2EAGK76E7ATO3Q, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:34.866655959 (571h53m34.737738994s) [noise=true , address=145.79.227.16:4910 , piece_id=MR4IES6CULVSVQHYOYSNHUL6RWDIXQHES2S43HPMFLCKR726MCIA, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:34.870192272 (571h53m34.741275317s) [noise=true , address=145.79.227.36:4519 , piece_id=CYHA5B5AAYLQYGG7PKDZBL6KBNL7PWAXC3XJESNZJZNG6VFGAUDQ, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:34.870875714 (571h53m34.741958749s) [noise=true , address=145.79.227.50:4374 , piece_id=PYOD755FKX2UEJ2KGDSO77M4I5MY2GUGZ6WPKC5P73QYRQXTOYLQ, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:34.872583666 (571h53m34.743666701s) [noise=true , address=145.79.227.28:4726 , piece_id=H5FAH27QEOC6PMAZSX7FVEIMGI6JKTQAOYKMBXSFXRYNQT25324A, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:34.890730350 (571h53m34.761813395s) [noise=true , address=145.79.227.51:4409 , piece_id=3K5GWDCXZD2EU2L4J7J57IWCSGXHEMPFVO4YXH4SUCBGAFXOY7GA, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:34.891395006 (571h53m34.762478051s) [noise=true , address=145.79.227.19:4079 , piece_id=JCRBTXMT3G6GJA5K4XLZW2Y6KSHX5JAXW5GJGH6IMMWL5F47MNRQ, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:34.895657111 (571h53m34.766740146s) [noise=true , address=145.79.227.6:4258 , piece_id=U6EG23QFBI3OR6ECAMRXZXZUWZ36RC5VY54LFXGWBEQN6PWNYBZA, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:35.227622103 (571h53m35.098705139s) [noise=true , address=45.91.116.5:4297 , piece_id=VMKLHK4Z34NW75AVI4ZA2OA7ONSTY4C2RIR6T4AGANRCGHBIP4GA, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:35.261404501 (571h53m35.132487556s) [noise=true , address=45.91.116.8:4045 , piece_id=HNWAPBAX36BBXIDGUPQDTDSQA4PZ4TGB22EUWV3RYGUHZCEZOE2Q, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:35.280892309 (571h53m35.151975344s) [noise=true , address=45.91.116.10:4084 , piece_id=4KU7BI4YDC62OWW452QQBBL6FPPQ4XFV6622H3S7NHVH3RC4QUNA, seg_pos={0 0}, upload=501383]: Done uploading piece. err: <nil> Dec 29 11:22:35.280914941 (571h53m35.151997986s) [seg_pos={0 0}, upload=501383]: Segment reached optimal threshold of 13 pieces. Dec 29 11:22:35.281224681 (571h53m35.152307746s) [noise=true , address=77.90.31.10:4517 , piece_id=3QAS444K45QVWONIQJAJVQLKQPKD62ER2NUOFUAMUO7LCF7WWPGQ, seg_pos={0 0}, upload=501383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:35.281234840 (571h53m35.152317885s) [noise=true , address=77.90.31.10:4517 , piece_id=3QAS444K45QVWONIQJAJVQLKQPKD62ER2NUOFUAMUO7LCF7WWPGQ, seg_pos={0 0}, upload=501383]: Long tail cancellation due to optimal threshold Dec 29 11:22:35.281401778 (571h53m35.152484823s) [noise=true , address=45.91.116.14:4190 , piece_id=4LEIJANQSVBZH3XCRNKZ6LBQBK3P46Q6DOFD2VKFVCCBEIL7ECEA, seg_pos={0 0}, upload=501383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:35.281424642 (571h53m35.152507677s) [noise=true , address=45.91.116.14:4190 , piece_id=4LEIJANQSVBZH3XCRNKZ6LBQBK3P46Q6DOFD2VKFVCCBEIL7ECEA, seg_pos={0 0}, upload=501383]: Long tail cancellation due to optimal threshold Dec 29 11:22:35.281569658 (571h53m35.152652703s) [noise=true , address=45.91.116.9:4264 , piece_id=O23BR37PP6JQXJVHRI65MNPHKXXBKEMXGEEKVKSFS3NJQIQEWX2Q, seg_pos={0 0}, upload=501383]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 29 11:22:35.281578355 (571h53m35.15266139s) [noise=true , address=45.91.116.9:4264 , piece_id=O23BR37PP6JQXJVHRI65MNPHKXXBKEMXGEEKVKSFS3NJQIQEWX2Q, seg_pos={0 0}, upload=501383]: Long tail cancellation due to optimal threshold Dec 29 11:22:35.281598103 (571h53m35.152681148s) [seg_pos={0 0}, upload=501383]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 11:22:35.281676261 (571h53m35.152759316s) [upload=501383]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:22:35.281682853 (571h53m35.152765889s) [upload=501383]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:22:35.572843998 (571h53m35.443927044s) [upload=501383]: Done uploading segments... Dec 29 11:22:35.572864308 (571h53m35.443947353s) [upload=501383]: Upload finished. err: <nil> Duration: 30.002806357s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [XX......X...X....X...X..........X....X.......X..X...XX..XX..] duration: 3.677305856s trace: https://jaeger.dev.storj.io/trace/6c77d78dac3d14a1 FAIL (2025-12-29 11:22:53 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.677332266s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X..................................X...............X..] duration: 44.643272116s trace: https://jaeger.dev.storj.io/trace/30604d8b334c422 FAIL (2025-12-29 11:22:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== s) [noise=true , address=77.90.31.11:4588 , piece_id=USOLW6Z6EEZLE2V5XA7N57XBKHQBERW5MXSWPWFWJQMYMWX7MENQ, seg_pos={0 0}, upload=501341]: Uploading piece... Dec 29 11:19:40.511735860 (571h50m40.382818905s) [noise=true , address=64.49.13.169:4713 , piece_id=G5ETFSQY3WMQH6RWX2XRDHQHDD73VSDUVUEXXBQJHE2RZOSY2L5A, seg_pos={0 0}, upload=501341]: Uploading piece... Dec 29 11:19:40.512126774 (571h50m40.38320982s) [noise=true , address=145.79.227.17:4980 , piece_id=AVU4L6PVWYLJV3D3KZKMG6ZVSYXJC5KVQ5HPUOCJRHBNLRJWA5SA, seg_pos={0 0}, upload=501341]: Uploading piece... Dec 29 11:19:40.512143016 (571h50m40.383226051s) [noise=true , address=145.79.227.34:4910 , piece_id=OI7WBL7EHIAJTLCMKOLBYJL3ZXBU3AHS7UWJOD3PSPJP3S6V4TBA, seg_pos={0 0}, upload=501341]: Uploading piece... Dec 29 11:19:40.512533459 (571h50m40.383616504s) [noise=true , address=145.79.227.43:4178 , piece_id=2BIXT4MN6DT3FTBLWFY5EKN4P5N5RZC2OARQK37WCZE7H4UAAN6A, seg_pos={0 0}, upload=501341]: Uploading piece... Dec 29 11:19:40.512555862 (571h50m40.383638897s) [noise=true , address=145.79.227.44:4205 , piece_id=J5M33FRANHDPN4W4BM23Q6OFQ2REJFVW6DO2LHAKGWJB6GBMTRNQ, seg_pos={0 0}, upload=501341]: Uploading piece... Dec 29 11:19:40.512925265 (571h50m40.38400831s) [noise=true , address=145.79.227.13:4831 , piece_id=YRURTAZFA7NGCRLHTENMVI2KJ6MMIDVFICY32KXDMORS4ZRYJ2AQ, seg_pos={0 0}, upload=501341]: Uploading piece... Dec 29 11:19:40.512947137 (571h50m40.384030172s) [noise=true , address=145.79.227.40:4060 , piece_id=TN5O2G72ZCAYGOUCZDI6JCJOKEUQXWFBE4GOHFXRRX7ENMAYAXZA, seg_pos={0 0}, upload=501341]: Uploading piece... Dec 29 11:19:40.513381343 (571h50m40.384464378s) [noise=true , address=77.90.31.21:4881 , piece_id=U4DNLO5JMNWLRYXPG5QCOMD5M5KM3SHNU53UFSQKLF76E3FVELNA, seg_pos={0 0}, upload=501341]: Uploading piece... Dec 29 11:19:40.513444153 (571h50m40.384527198s) [noise=true , address=145.79.227.30:4771 , piece_id=E6UTG5G4RB4F7ARBX4IR3P5NO24CALQ7E3H22KP5OCKZ3VL74ECA, seg_pos={0 0}, upload=501341]: Uploading piece... Dec 29 11:19:40.513788890 (571h50m40.384871925s) [noise=true , address=45.91.116.11:4147 , piece_id=GFGWBPMIIFSVET5TIPIX5RHWQ66SQLJS4WAAXV4S75QP7SAZYOBQ, seg_pos={0 0}, upload=501341]: Uploading piece... Dec 29 11:19:40.513965336 (571h50m40.385048381s) [noise=true , address=145.79.227.35:4513 , piece_id=XB2F2LAXXP54R3J7R23OOGFMJXAMXDRIWKZTVY2I2OUZG5NBS5PA, seg_pos={0 0}, upload=501341]: Uploading piece... Dec 29 11:19:40.514485947 (571h50m40.385568992s) [noise=true , address=145.79.227.12:4776 , piece_id=4VNNRBRGCHXGTVVU37FH4FYLEIKVIXTVPS4BAFT3EGM62IHGXFEA, seg_pos={0 0}, upload=501341]: Uploading piece... Dec 29 11:19:40.514891830 (571h50m40.385974865s) [noise=true , address=45.91.116.10:4094 , piece_id=DMNIOWGGVZFV47SAEPTAQAW7JECUGUQ2JVN7LF4QA23RANX42V3Q, seg_pos={0 0}, upload=501341]: Uploading piece... Dec 29 11:19:40.515312531 (571h50m40.386395576s) [noise=true , address=145.79.227.33:4903 , piece_id=CZPMPIAJV437JVEPTEJEASZDE2RBWOP7PXEHBPEYBVHD6J2LTFSA, seg_pos={0 0}, upload=501341]: Uploading piece... Dec 29 11:19:40.516926965 (571h50m40.38801001s) [noise=true , address=193.149.164.143:4704 , piece_id=XADBOYOBSRNEU6IEN7NCBFRRDGLHV7367V6ARDLENLUDMH76V6SQ, seg_pos={0 0}, upload=501341]: Uploading piece... Dec 29 11:19:41.844852478 (571h50m41.715935513s) [upload=501341]: Next returned nil segment Dec 29 11:19:41.844866866 (571h50m41.715949901s) [upload=501341]: Waiting for error group managing segments... Dec 29 11:19:42.375579856 (571h50m42.246662891s) [noise=true , address=145.79.227.12:4776 , piece_id=4VNNRBRGCHXGTVVU37FH4FYLEIKVIXTVPS4BAFT3EGM62IHGXFEA, seg_pos={0 0}, upload=501341]: Done uploading piece. err: <nil> Dec 29 11:19:42.376792365 (571h50m42.24787541s) [noise=true , address=145.79.227.17:4980 , piece_id=AVU4L6PVWYLJV3D3KZKMG6ZVSYXJC5KVQ5HPUOCJRHBNLRJWA5SA, seg_pos={0 0}, upload=501341]: Done uploading piece. err: <nil> Dec 29 11:19:42.378045631 (571h50m42.249128676s) [noise=true , address=145.79.227.13:4831 , piece_id=YRURTAZFA7NGCRLHTENMVI2KJ6MMIDVFICY32KXDMORS4ZRYJ2AQ, seg_pos={0 0}, upload=501341]: Done uploading piece. err: <nil> Dec 29 11:19:42.386717985 (571h50m42.25780102s) [noise=true , address=145.79.227.33:4903 , piece_id=CZPMPIAJV437JVEPTEJEASZDE2RBWOP7PXEHBPEYBVHD6J2LTFSA, seg_pos={0 0}, upload=501341]: Done uploading piece. err: <nil> Dec 29 11:19:42.403162488 (571h50m42.274245523s) [noise=true , address=145.79.227.35:4513 , piece_id=XB2F2LAXXP54R3J7R23OOGFMJXAMXDRIWKZTVY2I2OUZG5NBS5PA, seg_pos={0 0}, upload=501341]: Done uploading piece. err: <nil> Dec 29 11:19:42.669549168 (571h50m42.540632214s) [noise=true , address=77.90.31.16:4753 , piece_id=TAS7IRJLJJNFO35X6BTW7UOPSNGVLQEETBUIRI4QFP6ZFKNSN3DA, seg_pos={0 0}, upload=501341]: Done uploading piece. err: <nil> Dec 29 11:19:42.696929825 (571h50m42.56801286s) [noise=true , address=77.90.31.21:4881 , piece_id=U4DNLO5JMNWLRYXPG5QCOMD5M5KM3SHNU53UFSQKLF76E3FVELNA, seg_pos={0 0}, upload=501341]: Done uploading piece. err: <nil> Dec 29 11:19:43.153536604 (571h50m43.024619649s) [noise=true , address=145.79.227.40:4060 , piece_id=TN5O2G72ZCAYGOUCZDI6JCJOKEUQXWFBE4GOHFXRRX7ENMAYAXZA, seg_pos={0 0}, upload=501341]: Done uploading piece. err: <nil> Dec 29 11:19:43.234523837 (571h50m43.105606882s) [noise=true , address=145.79.227.44:4205 , piece_id=J5M33FRANHDPN4W4BM23Q6OFQ2REJFVW6DO2LHAKGWJB6GBMTRNQ, seg_pos={0 0}, upload=501341]: Done uploading piece. err: <nil> Dec 29 11:19:43.306005738 (571h50m43.177088773s) [noise=true , address=145.79.227.43:4178 , piece_id=2BIXT4MN6DT3FTBLWFY5EKN4P5N5RZC2OARQK37WCZE7H4UAAN6A, seg_pos={0 0}, upload=501341]: Done uploading piece. err: <nil> Dec 29 11:19:43.333711425 (571h50m43.20479446s) [noise=true , address=193.149.164.143:4702 , piece_id=YL7JDBVMKUUHZREXP2OEKCR573NC7RYKFG3BREKQN3Z3LBISJF7A, seg_pos={0 0}, upload=501341]: Done uploading piece. err: <nil> Dec 29 11:19:43.359543432 (571h50m43.230626477s) [noise=true , address=145.79.227.34:4910 , piece_id=OI7WBL7EHIAJTLCMKOLBYJL3ZXBU3AHS7UWJOD3PSPJP3S6V4TBA, seg_pos={0 0}, upload=501341]: Done uploading piece. err: <nil> Dec 29 11:19:43.394210304 (571h50m43.265293349s) [noise=true , address=145.79.227.30:4771 , piece_id=E6UTG5G4RB4F7ARBX4IR3P5NO24CALQ7E3H22KP5OCKZ3VL74ECA, seg_pos={0 0}, upload=501341]: Done uploading piece. err: <nil> Dec 29 11:19:43.448596804 (571h50m43.319679849s) [noise=true , address=45.91.116.10:4094 , piece_id=DMNIOWGGVZFV47SAEPTAQAW7JECUGUQ2JVN7LF4QA23RANX42V3Q, seg_pos={0 0}, upload=501341]: Done uploading piece. err: <nil> Dec 29 11:19:43.448626000 (571h50m43.319709045s) [seg_pos={0 0}, upload=501341]: Segment reached optimal threshold of 14 pieces. Dec 29 11:19:43.448914891 (571h50m43.319997936s) [noise=true , address=77.90.31.11:4588 , piece_id=USOLW6Z6EEZLE2V5XA7N57XBKHQBERW5MXSWPWFWJQMYMWX7MENQ, seg_pos={0 0}, upload=501341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6): protocol: expected piece hash; context canceled; EOF Dec 29 11:19:43.448926743 (571h50m43.320009788s) [noise=true , address=77.90.31.11:4588 , piece_id=USOLW6Z6EEZLE2V5XA7N57XBKHQBERW5MXSWPWFWJQMYMWX7MENQ, seg_pos={0 0}, upload=501341]: Long tail cancellation due to optimal threshold Dec 29 11:19:43.449142253 (571h50m43.320225298s) [noise=true , address=193.149.164.143:4704 , piece_id=XADBOYOBSRNEU6IEN7NCBFRRDGLHV7367V6ARDLENLUDMH76V6SQ, seg_pos={0 0}, upload=501341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 29 11:19:43.449151661 (571h50m43.320234706s) [noise=true , address=193.149.164.143:4704 , piece_id=XADBOYOBSRNEU6IEN7NCBFRRDGLHV7367V6ARDLENLUDMH76V6SQ, seg_pos={0 0}, upload=501341]: Long tail cancellation due to optimal threshold Dec 29 11:19:43.449288411 (571h50m43.320371456s) [noise=true , address=45.91.116.11:4147 , piece_id=GFGWBPMIIFSVET5TIPIX5RHWQ66SQLJS4WAAXV4S75QP7SAZYOBQ, seg_pos={0 0}, upload=501341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YnWetw1NereHQi7cwHCnmu1LDjRFNpB2MQHSWQrjQdN1CZbMw): protocol: expected piece hash; context canceled; EOF Dec 29 11:19:43.449296276 (571h50m43.320379321s) [noise=true , address=45.91.116.11:4147 , piece_id=GFGWBPMIIFSVET5TIPIX5RHWQ66SQLJS4WAAXV4S75QP7SAZYOBQ, seg_pos={0 0}, upload=501341]: Long tail cancellation due to optimal threshold Dec 29 11:19:43.449343737 (571h50m43.320426812s) [noise=true , address=64.49.13.169:4713 , piece_id=G5ETFSQY3WMQH6RWX2XRDHQHDD73VSDUVUEXXBQJHE2RZOSY2L5A, seg_pos={0 0}, upload=501341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:37770->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:37770->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37770->64.49.13.169:4713: use of closed network connection Dec 29 11:19:43.449355399 (571h50m43.320438454s) [noise=true , address=64.49.13.169:4713 , piece_id=G5ETFSQY3WMQH6RWX2XRDHQHDD73VSDUVUEXXBQJHE2RZOSY2L5A, seg_pos={0 0}, upload=501341]: Long tail cancellation due to optimal threshold Dec 29 11:19:43.449382020 (571h50m43.320465075s) [seg_pos={0 0}, upload=501341]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 11:19:43.449485477 (571h50m43.320568522s) [upload=501341]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:19:43.449494904 (571h50m43.32057795s) [upload=501341]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:19:43.738702540 (571h50m43.609785575s) [upload=501341]: Done uploading segments... Dec 29 11:19:43.738721196 (571h50m43.609804241s) [upload=501341]: Upload finished. err: <nil> Duration: 44.643289198s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [XXX.X..XXX..X.X.XX....X..X.....X.X...X.X.X.X.XX..XX..X..XXX.] duration: 5.956152949s trace: https://jaeger.dev.storj.io/trace/3150510ffcfb2c6d FAIL (2025-12-29 11:22:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: FBQPARU4HK7VNNYCKISIS634MNO4HUBDB2SQDVK555EBY7X4NF5Q) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: HSFPRE6H5JDJB5265E4RJIXMAX64XGZEOPTM7Z7Z2AXSRIYYAZYA) context deadline exceeded ======LOGS====== Dec 29 11:21:46.583273237 (571h52m46.454356282s) [upload=501370]: Starting upload Dec 29 11:21:46.583364831 (571h52m46.454447866s) [upload=501370]: Done starting upload Dec 29 11:21:46.583398606 (571h52m46.454481651s) [upload=501370]: Uploading segments... Dec 29 11:21:46.583642029 (571h52m46.454725074s) [upload=501370]: Got next segment. Inline: false Dec 29 11:21:46.583686754 (571h52m46.454769799s) [upload=501370]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 11:21:46.583692315 (571h52m46.45477536s) [upload=501370]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 11:21:46.830995109 (571h52m46.702078144s) [seg_pos={0 0}, upload=501370]: Begin upload segment... Dec 29 11:21:46.831035175 (571h52m46.70211821s) [seg_pos={0 0}, upload=501370]: Done begin upload segment. Dec 29 11:21:46.831062587 (571h52m46.702145622s) [noise=true , address=77.90.31.10:4518 , piece_id=HSFPRE6H5JDJB5265E4RJIXMAX64XGZEOPTM7Z7Z2AXSRIYYAZYA, seg_pos={0 0}, upload=501370]: Uploading piece... Dec 29 11:21:46.831116189 (571h52m46.702199224s) [noise=true , address=45.91.116.5:4300 , piece_id=3MP27CBHODZYKASW2LEEOHC22EHIO3AYPX6UFCO7ZD3IWN42LTFQ, seg_pos={0 0}, upload=501370]: Uploading piece... Dec 29 11:21:46.831633865 (571h52m46.7027169s) [noise=true , address=45.91.116.8:4045 , piece_id=QHFHM3BRN22QRCZIPNJHH5JFMSHE3QAIWHPCIHR4VZTI2LLFIV3A, seg_pos={0 0}, upload=501370]: Uploading piece... Dec 29 11:21:46.831772859 (571h52m46.702855894s) [noise=true , address=145.79.227.34:4906 , piece_id=W36ALE7K7URWH36OQVTDXRTAFQMIKY7VX7JE33JCEKWY5Y6YHACQ, seg_pos={0 0}, upload=501370]: Uploading piece... Dec 29 11:21:46.832208058 (571h52m46.703291093s) [noise=true , address=145.79.227.4:4186 , piece_id=46M2YC4UYUWBJWUTOL4DQM44TDMNWXOE2WORDDTIYOCH5KIXDT2A, seg_pos={0 0}, upload=501370]: Uploading piece... Dec 29 11:21:46.832859449 (571h52m46.703942494s) [noise=true , address=45.91.116.7:4010 , piece_id=H3DRA6VM6VRJXBW4YZOFQHTTVOFLS5BCGCTT2K7GSSG4HOKR2AOA, seg_pos={0 0}, upload=501370]: Uploading piece... Dec 29 11:21:46.833373467 (571h52m46.704456512s) [noise=true , address=145.79.227.6:4257 , piece_id=BDPRIOTKD6XUA3CRQ72XYYJWBF6ME6YOXKBFVL6EQHJ6EV42DDUQ, seg_pos={0 0}, upload=501370]: Uploading piece... Dec 29 11:21:46.840313603 (571h52m46.711396638s) [noise=true , address=145.79.227.2:4114 , piece_id=GLUNT2NPW4Q6DTJZETHXCL2BUJQJEL2MMNXO2MVL2PVXQ5P6MIZA, seg_pos={0 0}, upload=501370]: Uploading piece... Dec 29 11:21:46.841569023 (571h52m46.712652068s) [noise=true , address=145.79.227.7:4295 , piece_id=FKUCQ52324QGO7U7LNOXBQPZ6VUYR4RT6DI67P2OZNLFVJ4OO44A, seg_pos={0 0}, upload=501370]: Uploading piece... Dec 29 11:21:46.843453291 (571h52m46.714536326s) [noise=true , address=145.79.227.16:4910 , piece_id=YDPI6IZUQRQJOZCSWBKAVOSGKN2SOSCCSOFM6VASUZKESLGMVASQ, seg_pos={0 0}, upload=501370]: Uploading piece... Dec 29 11:21:46.844269605 (571h52m46.71535264s) [noise=true , address=145.79.227.41:4085 , piece_id=ADO4KZOHCGEETEPMIPJDBK2CIOHXF33GISJJQEOPNPCH3RXXL34A, seg_pos={0 0}, upload=501370]: Uploading piece... Dec 29 11:21:46.845341386 (571h52m46.716424421s) [noise=true , address=145.79.227.39:4013 , piece_id=ZGICQOMHWP4AZ72ZY3X4GPZLEWZMZDOEMHEBHDVUAH3KTDK3LENA, seg_pos={0 0}, upload=501370]: Uploading piece... Dec 29 11:21:46.846121552 (571h52m46.717204588s) [noise=true , address=145.79.227.5:4221 , piece_id=ABO76JLREGJ2GEL44Y66RPSATRFNASTABTPEFG4D2B64N7LAHNAQ, seg_pos={0 0}, upload=501370]: Uploading piece... Dec 29 11:21:46.847827050 (571h52m46.718910085s) [noise=true , address=145.79.227.28:4726 , piece_id=3377M4VSPBLR2QD5FD55VLOSBZVVWI66PVKSWVZ3LPBVB652XO5Q, seg_pos={0 0}, upload=501370]: Uploading piece... Dec 29 11:21:46.848149384 (571h52m46.719232419s) [noise=true , address=45.91.116.9:4262 , piece_id=FBQPARU4HK7VNNYCKISIS634MNO4HUBDB2SQDVK555EBY7X4NF5Q, seg_pos={0 0}, upload=501370]: Uploading piece... Dec 29 11:21:46.848183759 (571h52m46.719266794s) [noise=true , address=45.91.116.6:4974 , piece_id=TE343JAIJWFRSYXM4LAOPC7CJ6CL3B3WI3B2IRJ5SYIJ2JKPECHA, seg_pos={0 0}, upload=501370]: Uploading piece... Dec 29 11:21:48.534690238 (571h52m48.405773273s) [upload=501370]: Next returned nil segment Dec 29 11:21:48.534710305 (571h52m48.40579335s) [upload=501370]: Waiting for error group managing segments... Dec 29 11:21:48.819198566 (571h52m48.690281611s) [noise=true , address=145.79.227.6:4257 , piece_id=BDPRIOTKD6XUA3CRQ72XYYJWBF6ME6YOXKBFVL6EQHJ6EV42DDUQ, seg_pos={0 0}, upload=501370]: Done uploading piece. err: <nil> Dec 29 11:21:48.830171961 (571h52m48.701255006s) [noise=true , address=145.79.227.4:4186 , piece_id=46M2YC4UYUWBJWUTOL4DQM44TDMNWXOE2WORDDTIYOCH5KIXDT2A, seg_pos={0 0}, upload=501370]: Done uploading piece. err: <nil> Dec 29 11:21:48.831775614 (571h52m48.702858659s) [noise=true , address=145.79.227.41:4085 , piece_id=ADO4KZOHCGEETEPMIPJDBK2CIOHXF33GISJJQEOPNPCH3RXXL34A, seg_pos={0 0}, upload=501370]: Done uploading piece. err: <nil> Dec 29 11:21:48.833643812 (571h52m48.704726847s) [noise=true , address=145.79.227.39:4013 , piece_id=ZGICQOMHWP4AZ72ZY3X4GPZLEWZMZDOEMHEBHDVUAH3KTDK3LENA, seg_pos={0 0}, upload=501370]: Done uploading piece. err: <nil> Dec 29 11:21:48.838183666 (571h52m48.709266701s) [noise=true , address=145.79.227.16:4910 , piece_id=YDPI6IZUQRQJOZCSWBKAVOSGKN2SOSCCSOFM6VASUZKESLGMVASQ, seg_pos={0 0}, upload=501370]: Done uploading piece. err: <nil> Dec 29 11:21:48.838865725 (571h52m48.70994876s) [noise=true , address=145.79.227.5:4221 , piece_id=ABO76JLREGJ2GEL44Y66RPSATRFNASTABTPEFG4D2B64N7LAHNAQ, seg_pos={0 0}, upload=501370]: Done uploading piece. err: <nil> Dec 29 11:21:48.841300270 (571h52m48.712383315s) [noise=true , address=145.79.227.28:4726 , piece_id=3377M4VSPBLR2QD5FD55VLOSBZVVWI66PVKSWVZ3LPBVB652XO5Q, seg_pos={0 0}, upload=501370]: Done uploading piece. err: <nil> Dec 29 11:21:48.857514064 (571h52m48.728597099s) [noise=true , address=145.79.227.2:4114 , piece_id=GLUNT2NPW4Q6DTJZETHXCL2BUJQJEL2MMNXO2MVL2PVXQ5P6MIZA, seg_pos={0 0}, upload=501370]: Done uploading piece. err: <nil> Dec 29 11:21:48.860717123 (571h52m48.731800169s) [noise=true , address=145.79.227.7:4295 , piece_id=FKUCQ52324QGO7U7LNOXBQPZ6VUYR4RT6DI67P2OZNLFVJ4OO44A, seg_pos={0 0}, upload=501370]: Done uploading piece. err: <nil> Dec 29 11:21:49.103261614 (571h52m48.974344649s) [noise=true , address=45.91.116.8:4045 , piece_id=QHFHM3BRN22QRCZIPNJHH5JFMSHE3QAIWHPCIHR4VZTI2LLFIV3A, seg_pos={0 0}, upload=501370]: Done uploading piece. err: <nil> Dec 29 11:21:49.111062829 (571h52m48.982145864s) [noise=true , address=45.91.116.6:4974 , piece_id=TE343JAIJWFRSYXM4LAOPC7CJ6CL3B3WI3B2IRJ5SYIJ2JKPECHA, seg_pos={0 0}, upload=501370]: Done uploading piece. err: <nil> Dec 29 11:21:49.136234790 (571h52m49.007317835s) [noise=true , address=45.91.116.9:4262 , piece_id=FBQPARU4HK7VNNYCKISIS634MNO4HUBDB2SQDVK555EBY7X4NF5Q, seg_pos={0 0}, upload=501370]: Done uploading piece. err: <nil> Dec 29 11:21:49.136782203 (571h52m49.007865238s) [noise=true , address=77.90.31.10:4518 , piece_id=HSFPRE6H5JDJB5265E4RJIXMAX64XGZEOPTM7Z7Z2AXSRIYYAZYA, seg_pos={0 0}, upload=501370]: Done uploading piece. err: <nil> Dec 29 11:21:49.136792412 (571h52m49.007875447s) [seg_pos={0 0}, upload=501370]: Segment reached optimal threshold of 13 pieces. Dec 29 11:21:49.136979979 (571h52m49.008063015s) [noise=true , address=145.79.227.34:4906 , piece_id=W36ALE7K7URWH36OQVTDXRTAFQMIKY7VX7JE33JCEKWY5Y6YHACQ, seg_pos={0 0}, upload=501370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 29 11:21:49.136988285 (571h52m49.00807132s) [noise=true , address=145.79.227.34:4906 , piece_id=W36ALE7K7URWH36OQVTDXRTAFQMIKY7VX7JE33JCEKWY5Y6YHACQ, seg_pos={0 0}, upload=501370]: Long tail cancellation due to optimal threshold Dec 29 11:21:49.137401382 (571h52m49.008484427s) [noise=true , address=45.91.116.7:4010 , piece_id=H3DRA6VM6VRJXBW4YZOFQHTTVOFLS5BCGCTT2K7GSSG4HOKR2AOA, seg_pos={0 0}, upload=501370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 29 11:21:49.137410238 (571h52m49.008493284s) [noise=true , address=45.91.116.7:4010 , piece_id=H3DRA6VM6VRJXBW4YZOFQHTTVOFLS5BCGCTT2K7GSSG4HOKR2AOA, seg_pos={0 0}, upload=501370]: Long tail cancellation due to optimal threshold Dec 29 11:21:49.137559392 (571h52m49.008642437s) [noise=true , address=45.91.116.5:4300 , piece_id=3MP27CBHODZYKASW2LEEOHC22EHIO3AYPX6UFCO7ZD3IWN42LTFQ, seg_pos={0 0}, upload=501370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 29 11:21:49.137567148 (571h52m49.008650183s) [noise=true , address=45.91.116.5:4300 , piece_id=3MP27CBHODZYKASW2LEEOHC22EHIO3AYPX6UFCO7ZD3IWN42LTFQ, seg_pos={0 0}, upload=501370]: Long tail cancellation due to optimal threshold Dec 29 11:21:49.137584941 (571h52m49.008667986s) [seg_pos={0 0}, upload=501370]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 11:21:49.137664422 (571h52m49.008747467s) [upload=501370]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:21:49.137671597 (571h52m49.008754632s) [upload=501370]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:21:49.425521617 (571h52m49.296604652s) [upload=501370]: Done uploading segments... Dec 29 11:21:49.425541895 (571h52m49.29662493s) [upload=501370]: Upload finished. err: <nil> Duration: 5.956170503s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [.XXX.X..XXX..X.X.XX....X..X.....X.X...X.X.X.X.XX..XX..X..XXX] duration: 30.002834791s trace: https://jaeger.dev.storj.io/trace/3150510ffcfb2c6d FAIL (2025-12-29 11:22:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: FBQPARU4HK7VNNYCKISIS634MNO4HUBDB2SQDVK555EBY7X4NF5Q) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: HSFPRE6H5JDJB5265E4RJIXMAX64XGZEOPTM7Z7Z2AXSRIYYAZYA) context deadline exceeded ======LOGS====== Dec 29 11:21:46.583273237 (571h52m46.454356282s) [upload=501370]: Starting upload Dec 29 11:21:46.583364831 (571h52m46.454447866s) [upload=501370]: Done starting upload Dec 29 11:21:46.583398606 (571h52m46.454481651s) [upload=501370]: Uploading segments... Dec 29 11:21:46.583642029 (571h52m46.454725074s) [upload=501370]: Got next segment. Inline: false Dec 29 11:21:46.583686754 (571h52m46.454769799s) [upload=501370]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 11:21:46.583692315 (571h52m46.45477536s) [upload=501370]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 11:21:46.830995109 (571h52m46.702078144s) [seg_pos={0 0}, upload=501370]: Begin upload segment... Dec 29 11:21:46.831035175 (571h52m46.70211821s) [seg_pos={0 0}, upload=501370]: Done begin upload segment. Dec 29 11:21:46.831062587 (571h52m46.702145622s) [noise=true , address=77.90.31.10:4518 , piece_id=HSFPRE6H5JDJB5265E4RJIXMAX64XGZEOPTM7Z7Z2AXSRIYYAZYA, seg_pos={0 0}, upload=501370]: Uploading piece... Dec 29 11:21:46.831116189 (571h52m46.702199224s) [noise=true , address=45.91.116.5:4300 , piece_id=3MP27CBHODZYKASW2LEEOHC22EHIO3AYPX6UFCO7ZD3IWN42LTFQ, seg_pos={0 0}, upload=501370]: Uploading piece... Dec 29 11:21:46.831633865 (571h52m46.7027169s) [noise=true , address=45.91.116.8:4045 , piece_id=QHFHM3BRN22QRCZIPNJHH5JFMSHE3QAIWHPCIHR4VZTI2LLFIV3A, seg_pos={0 0}, upload=501370]: Uploading piece... Dec 29 11:21:46.831772859 (571h52m46.702855894s) [noise=true , address=145.79.227.34:4906 , piece_id=W36ALE7K7URWH36OQVTDXRTAFQMIKY7VX7JE33JCEKWY5Y6YHACQ, seg_pos={0 0}, upload=501370]: Uploading piece... Dec 29 11:21:46.832208058 (571h52m46.703291093s) [noise=true , address=145.79.227.4:4186 , piece_id=46M2YC4UYUWBJWUTOL4DQM44TDMNWXOE2WORDDTIYOCH5KIXDT2A, seg_pos={0 0}, upload=501370]: Uploading piece... Dec 29 11:21:46.832859449 (571h52m46.703942494s) [noise=true , address=45.91.116.7:4010 , piece_id=H3DRA6VM6VRJXBW4YZOFQHTTVOFLS5BCGCTT2K7GSSG4HOKR2AOA, seg_pos={0 0}, upload=501370]: Uploading piece... Dec 29 11:21:46.833373467 (571h52m46.704456512s) [noise=true , address=145.79.227.6:4257 , piece_id=BDPRIOTKD6XUA3CRQ72XYYJWBF6ME6YOXKBFVL6EQHJ6EV42DDUQ, seg_pos={0 0}, upload=501370]: Uploading piece... Dec 29 11:21:46.840313603 (571h52m46.711396638s) [noise=true , address=145.79.227.2:4114 , piece_id=GLUNT2NPW4Q6DTJZETHXCL2BUJQJEL2MMNXO2MVL2PVXQ5P6MIZA, seg_pos={0 0}, upload=501370]: Uploading piece... Dec 29 11:21:46.841569023 (571h52m46.712652068s) [noise=true , address=145.79.227.7:4295 , piece_id=FKUCQ52324QGO7U7LNOXBQPZ6VUYR4RT6DI67P2OZNLFVJ4OO44A, seg_pos={0 0}, upload=501370]: Uploading piece... Dec 29 11:21:46.843453291 (571h52m46.714536326s) [noise=true , address=145.79.227.16:4910 , piece_id=YDPI6IZUQRQJOZCSWBKAVOSGKN2SOSCCSOFM6VASUZKESLGMVASQ, seg_pos={0 0}, upload=501370]: Uploading piece... Dec 29 11:21:46.844269605 (571h52m46.71535264s) [noise=true , address=145.79.227.41:4085 , piece_id=ADO4KZOHCGEETEPMIPJDBK2CIOHXF33GISJJQEOPNPCH3RXXL34A, seg_pos={0 0}, upload=501370]: Uploading piece... Dec 29 11:21:46.845341386 (571h52m46.716424421s) [noise=true , address=145.79.227.39:4013 , piece_id=ZGICQOMHWP4AZ72ZY3X4GPZLEWZMZDOEMHEBHDVUAH3KTDK3LENA, seg_pos={0 0}, upload=501370]: Uploading piece... Dec 29 11:21:46.846121552 (571h52m46.717204588s) [noise=true , address=145.79.227.5:4221 , piece_id=ABO76JLREGJ2GEL44Y66RPSATRFNASTABTPEFG4D2B64N7LAHNAQ, seg_pos={0 0}, upload=501370]: Uploading piece... Dec 29 11:21:46.847827050 (571h52m46.718910085s) [noise=true , address=145.79.227.28:4726 , piece_id=3377M4VSPBLR2QD5FD55VLOSBZVVWI66PVKSWVZ3LPBVB652XO5Q, seg_pos={0 0}, upload=501370]: Uploading piece... Dec 29 11:21:46.848149384 (571h52m46.719232419s) [noise=true , address=45.91.116.9:4262 , piece_id=FBQPARU4HK7VNNYCKISIS634MNO4HUBDB2SQDVK555EBY7X4NF5Q, seg_pos={0 0}, upload=501370]: Uploading piece... Dec 29 11:21:46.848183759 (571h52m46.719266794s) [noise=true , address=45.91.116.6:4974 , piece_id=TE343JAIJWFRSYXM4LAOPC7CJ6CL3B3WI3B2IRJ5SYIJ2JKPECHA, seg_pos={0 0}, upload=501370]: Uploading piece... Dec 29 11:21:48.534690238 (571h52m48.405773273s) [upload=501370]: Next returned nil segment Dec 29 11:21:48.534710305 (571h52m48.40579335s) [upload=501370]: Waiting for error group managing segments... Dec 29 11:21:48.819198566 (571h52m48.690281611s) [noise=true , address=145.79.227.6:4257 , piece_id=BDPRIOTKD6XUA3CRQ72XYYJWBF6ME6YOXKBFVL6EQHJ6EV42DDUQ, seg_pos={0 0}, upload=501370]: Done uploading piece. err: <nil> Dec 29 11:21:48.830171961 (571h52m48.701255006s) [noise=true , address=145.79.227.4:4186 , piece_id=46M2YC4UYUWBJWUTOL4DQM44TDMNWXOE2WORDDTIYOCH5KIXDT2A, seg_pos={0 0}, upload=501370]: Done uploading piece. err: <nil> Dec 29 11:21:48.831775614 (571h52m48.702858659s) [noise=true , address=145.79.227.41:4085 , piece_id=ADO4KZOHCGEETEPMIPJDBK2CIOHXF33GISJJQEOPNPCH3RXXL34A, seg_pos={0 0}, upload=501370]: Done uploading piece. err: <nil> Dec 29 11:21:48.833643812 (571h52m48.704726847s) [noise=true , address=145.79.227.39:4013 , piece_id=ZGICQOMHWP4AZ72ZY3X4GPZLEWZMZDOEMHEBHDVUAH3KTDK3LENA, seg_pos={0 0}, upload=501370]: Done uploading piece. err: <nil> Dec 29 11:21:48.838183666 (571h52m48.709266701s) [noise=true , address=145.79.227.16:4910 , piece_id=YDPI6IZUQRQJOZCSWBKAVOSGKN2SOSCCSOFM6VASUZKESLGMVASQ, seg_pos={0 0}, upload=501370]: Done uploading piece. err: <nil> Dec 29 11:21:48.838865725 (571h52m48.70994876s) [noise=true , address=145.79.227.5:4221 , piece_id=ABO76JLREGJ2GEL44Y66RPSATRFNASTABTPEFG4D2B64N7LAHNAQ, seg_pos={0 0}, upload=501370]: Done uploading piece. err: <nil> Dec 29 11:21:48.841300270 (571h52m48.712383315s) [noise=true , address=145.79.227.28:4726 , piece_id=3377M4VSPBLR2QD5FD55VLOSBZVVWI66PVKSWVZ3LPBVB652XO5Q, seg_pos={0 0}, upload=501370]: Done uploading piece. err: <nil> Dec 29 11:21:48.857514064 (571h52m48.728597099s) [noise=true , address=145.79.227.2:4114 , piece_id=GLUNT2NPW4Q6DTJZETHXCL2BUJQJEL2MMNXO2MVL2PVXQ5P6MIZA, seg_pos={0 0}, upload=501370]: Done uploading piece. err: <nil> Dec 29 11:21:48.860717123 (571h52m48.731800169s) [noise=true , address=145.79.227.7:4295 , piece_id=FKUCQ52324QGO7U7LNOXBQPZ6VUYR4RT6DI67P2OZNLFVJ4OO44A, seg_pos={0 0}, upload=501370]: Done uploading piece. err: <nil> Dec 29 11:21:49.103261614 (571h52m48.974344649s) [noise=true , address=45.91.116.8:4045 , piece_id=QHFHM3BRN22QRCZIPNJHH5JFMSHE3QAIWHPCIHR4VZTI2LLFIV3A, seg_pos={0 0}, upload=501370]: Done uploading piece. err: <nil> Dec 29 11:21:49.111062829 (571h52m48.982145864s) [noise=true , address=45.91.116.6:4974 , piece_id=TE343JAIJWFRSYXM4LAOPC7CJ6CL3B3WI3B2IRJ5SYIJ2JKPECHA, seg_pos={0 0}, upload=501370]: Done uploading piece. err: <nil> Dec 29 11:21:49.136234790 (571h52m49.007317835s) [noise=true , address=45.91.116.9:4262 , piece_id=FBQPARU4HK7VNNYCKISIS634MNO4HUBDB2SQDVK555EBY7X4NF5Q, seg_pos={0 0}, upload=501370]: Done uploading piece. err: <nil> Dec 29 11:21:49.136782203 (571h52m49.007865238s) [noise=true , address=77.90.31.10:4518 , piece_id=HSFPRE6H5JDJB5265E4RJIXMAX64XGZEOPTM7Z7Z2AXSRIYYAZYA, seg_pos={0 0}, upload=501370]: Done uploading piece. err: <nil> Dec 29 11:21:49.136792412 (571h52m49.007875447s) [seg_pos={0 0}, upload=501370]: Segment reached optimal threshold of 13 pieces. Dec 29 11:21:49.136979979 (571h52m49.008063015s) [noise=true , address=145.79.227.34:4906 , piece_id=W36ALE7K7URWH36OQVTDXRTAFQMIKY7VX7JE33JCEKWY5Y6YHACQ, seg_pos={0 0}, upload=501370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 29 11:21:49.136988285 (571h52m49.00807132s) [noise=true , address=145.79.227.34:4906 , piece_id=W36ALE7K7URWH36OQVTDXRTAFQMIKY7VX7JE33JCEKWY5Y6YHACQ, seg_pos={0 0}, upload=501370]: Long tail cancellation due to optimal threshold Dec 29 11:21:49.137401382 (571h52m49.008484427s) [noise=true , address=45.91.116.7:4010 , piece_id=H3DRA6VM6VRJXBW4YZOFQHTTVOFLS5BCGCTT2K7GSSG4HOKR2AOA, seg_pos={0 0}, upload=501370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 29 11:21:49.137410238 (571h52m49.008493284s) [noise=true , address=45.91.116.7:4010 , piece_id=H3DRA6VM6VRJXBW4YZOFQHTTVOFLS5BCGCTT2K7GSSG4HOKR2AOA, seg_pos={0 0}, upload=501370]: Long tail cancellation due to optimal threshold Dec 29 11:21:49.137559392 (571h52m49.008642437s) [noise=true , address=45.91.116.5:4300 , piece_id=3MP27CBHODZYKASW2LEEOHC22EHIO3AYPX6UFCO7ZD3IWN42LTFQ, seg_pos={0 0}, upload=501370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 29 11:21:49.137567148 (571h52m49.008650183s) [noise=true , address=45.91.116.5:4300 , piece_id=3MP27CBHODZYKASW2LEEOHC22EHIO3AYPX6UFCO7ZD3IWN42LTFQ, seg_pos={0 0}, upload=501370]: Long tail cancellation due to optimal threshold Dec 29 11:21:49.137584941 (571h52m49.008667986s) [seg_pos={0 0}, upload=501370]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 11:21:49.137664422 (571h52m49.008747467s) [upload=501370]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:21:49.137671597 (571h52m49.008754632s) [upload=501370]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:21:49.425521617 (571h52m49.296604652s) [upload=501370]: Done uploading segments... Dec 29 11:21:49.425541895 (571h52m49.29662493s) [upload=501370]: Upload finished. err: <nil> Duration: 30.00285511s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [XXX......X...X....X...X..........X....X.......X..X...XX..XX.] duration: 3.557530177s trace: https://jaeger.dev.storj.io/trace/6c77d78dac3d14a1 FAIL (2025-12-29 11:21:47 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.55755288s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......X..................................X...............X.] duration: 48.244509905s trace: https://jaeger.dev.storj.io/trace/30604d8b334c422 FAIL (2025-12-29 11:21:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== s) [noise=true , address=77.90.31.11:4588 , piece_id=USOLW6Z6EEZLE2V5XA7N57XBKHQBERW5MXSWPWFWJQMYMWX7MENQ, seg_pos={0 0}, upload=501341]: Uploading piece... Dec 29 11:19:40.511735860 (571h50m40.382818905s) [noise=true , address=64.49.13.169:4713 , piece_id=G5ETFSQY3WMQH6RWX2XRDHQHDD73VSDUVUEXXBQJHE2RZOSY2L5A, seg_pos={0 0}, upload=501341]: Uploading piece... Dec 29 11:19:40.512126774 (571h50m40.38320982s) [noise=true , address=145.79.227.17:4980 , piece_id=AVU4L6PVWYLJV3D3KZKMG6ZVSYXJC5KVQ5HPUOCJRHBNLRJWA5SA, seg_pos={0 0}, upload=501341]: Uploading piece... Dec 29 11:19:40.512143016 (571h50m40.383226051s) [noise=true , address=145.79.227.34:4910 , piece_id=OI7WBL7EHIAJTLCMKOLBYJL3ZXBU3AHS7UWJOD3PSPJP3S6V4TBA, seg_pos={0 0}, upload=501341]: Uploading piece... Dec 29 11:19:40.512533459 (571h50m40.383616504s) [noise=true , address=145.79.227.43:4178 , piece_id=2BIXT4MN6DT3FTBLWFY5EKN4P5N5RZC2OARQK37WCZE7H4UAAN6A, seg_pos={0 0}, upload=501341]: Uploading piece... Dec 29 11:19:40.512555862 (571h50m40.383638897s) [noise=true , address=145.79.227.44:4205 , piece_id=J5M33FRANHDPN4W4BM23Q6OFQ2REJFVW6DO2LHAKGWJB6GBMTRNQ, seg_pos={0 0}, upload=501341]: Uploading piece... Dec 29 11:19:40.512925265 (571h50m40.38400831s) [noise=true , address=145.79.227.13:4831 , piece_id=YRURTAZFA7NGCRLHTENMVI2KJ6MMIDVFICY32KXDMORS4ZRYJ2AQ, seg_pos={0 0}, upload=501341]: Uploading piece... Dec 29 11:19:40.512947137 (571h50m40.384030172s) [noise=true , address=145.79.227.40:4060 , piece_id=TN5O2G72ZCAYGOUCZDI6JCJOKEUQXWFBE4GOHFXRRX7ENMAYAXZA, seg_pos={0 0}, upload=501341]: Uploading piece... Dec 29 11:19:40.513381343 (571h50m40.384464378s) [noise=true , address=77.90.31.21:4881 , piece_id=U4DNLO5JMNWLRYXPG5QCOMD5M5KM3SHNU53UFSQKLF76E3FVELNA, seg_pos={0 0}, upload=501341]: Uploading piece... Dec 29 11:19:40.513444153 (571h50m40.384527198s) [noise=true , address=145.79.227.30:4771 , piece_id=E6UTG5G4RB4F7ARBX4IR3P5NO24CALQ7E3H22KP5OCKZ3VL74ECA, seg_pos={0 0}, upload=501341]: Uploading piece... Dec 29 11:19:40.513788890 (571h50m40.384871925s) [noise=true , address=45.91.116.11:4147 , piece_id=GFGWBPMIIFSVET5TIPIX5RHWQ66SQLJS4WAAXV4S75QP7SAZYOBQ, seg_pos={0 0}, upload=501341]: Uploading piece... Dec 29 11:19:40.513965336 (571h50m40.385048381s) [noise=true , address=145.79.227.35:4513 , piece_id=XB2F2LAXXP54R3J7R23OOGFMJXAMXDRIWKZTVY2I2OUZG5NBS5PA, seg_pos={0 0}, upload=501341]: Uploading piece... Dec 29 11:19:40.514485947 (571h50m40.385568992s) [noise=true , address=145.79.227.12:4776 , piece_id=4VNNRBRGCHXGTVVU37FH4FYLEIKVIXTVPS4BAFT3EGM62IHGXFEA, seg_pos={0 0}, upload=501341]: Uploading piece... Dec 29 11:19:40.514891830 (571h50m40.385974865s) [noise=true , address=45.91.116.10:4094 , piece_id=DMNIOWGGVZFV47SAEPTAQAW7JECUGUQ2JVN7LF4QA23RANX42V3Q, seg_pos={0 0}, upload=501341]: Uploading piece... Dec 29 11:19:40.515312531 (571h50m40.386395576s) [noise=true , address=145.79.227.33:4903 , piece_id=CZPMPIAJV437JVEPTEJEASZDE2RBWOP7PXEHBPEYBVHD6J2LTFSA, seg_pos={0 0}, upload=501341]: Uploading piece... Dec 29 11:19:40.516926965 (571h50m40.38801001s) [noise=true , address=193.149.164.143:4704 , piece_id=XADBOYOBSRNEU6IEN7NCBFRRDGLHV7367V6ARDLENLUDMH76V6SQ, seg_pos={0 0}, upload=501341]: Uploading piece... Dec 29 11:19:41.844852478 (571h50m41.715935513s) [upload=501341]: Next returned nil segment Dec 29 11:19:41.844866866 (571h50m41.715949901s) [upload=501341]: Waiting for error group managing segments... Dec 29 11:19:42.375579856 (571h50m42.246662891s) [noise=true , address=145.79.227.12:4776 , piece_id=4VNNRBRGCHXGTVVU37FH4FYLEIKVIXTVPS4BAFT3EGM62IHGXFEA, seg_pos={0 0}, upload=501341]: Done uploading piece. err: <nil> Dec 29 11:19:42.376792365 (571h50m42.24787541s) [noise=true , address=145.79.227.17:4980 , piece_id=AVU4L6PVWYLJV3D3KZKMG6ZVSYXJC5KVQ5HPUOCJRHBNLRJWA5SA, seg_pos={0 0}, upload=501341]: Done uploading piece. err: <nil> Dec 29 11:19:42.378045631 (571h50m42.249128676s) [noise=true , address=145.79.227.13:4831 , piece_id=YRURTAZFA7NGCRLHTENMVI2KJ6MMIDVFICY32KXDMORS4ZRYJ2AQ, seg_pos={0 0}, upload=501341]: Done uploading piece. err: <nil> Dec 29 11:19:42.386717985 (571h50m42.25780102s) [noise=true , address=145.79.227.33:4903 , piece_id=CZPMPIAJV437JVEPTEJEASZDE2RBWOP7PXEHBPEYBVHD6J2LTFSA, seg_pos={0 0}, upload=501341]: Done uploading piece. err: <nil> Dec 29 11:19:42.403162488 (571h50m42.274245523s) [noise=true , address=145.79.227.35:4513 , piece_id=XB2F2LAXXP54R3J7R23OOGFMJXAMXDRIWKZTVY2I2OUZG5NBS5PA, seg_pos={0 0}, upload=501341]: Done uploading piece. err: <nil> Dec 29 11:19:42.669549168 (571h50m42.540632214s) [noise=true , address=77.90.31.16:4753 , piece_id=TAS7IRJLJJNFO35X6BTW7UOPSNGVLQEETBUIRI4QFP6ZFKNSN3DA, seg_pos={0 0}, upload=501341]: Done uploading piece. err: <nil> Dec 29 11:19:42.696929825 (571h50m42.56801286s) [noise=true , address=77.90.31.21:4881 , piece_id=U4DNLO5JMNWLRYXPG5QCOMD5M5KM3SHNU53UFSQKLF76E3FVELNA, seg_pos={0 0}, upload=501341]: Done uploading piece. err: <nil> Dec 29 11:19:43.153536604 (571h50m43.024619649s) [noise=true , address=145.79.227.40:4060 , piece_id=TN5O2G72ZCAYGOUCZDI6JCJOKEUQXWFBE4GOHFXRRX7ENMAYAXZA, seg_pos={0 0}, upload=501341]: Done uploading piece. err: <nil> Dec 29 11:19:43.234523837 (571h50m43.105606882s) [noise=true , address=145.79.227.44:4205 , piece_id=J5M33FRANHDPN4W4BM23Q6OFQ2REJFVW6DO2LHAKGWJB6GBMTRNQ, seg_pos={0 0}, upload=501341]: Done uploading piece. err: <nil> Dec 29 11:19:43.306005738 (571h50m43.177088773s) [noise=true , address=145.79.227.43:4178 , piece_id=2BIXT4MN6DT3FTBLWFY5EKN4P5N5RZC2OARQK37WCZE7H4UAAN6A, seg_pos={0 0}, upload=501341]: Done uploading piece. err: <nil> Dec 29 11:19:43.333711425 (571h50m43.20479446s) [noise=true , address=193.149.164.143:4702 , piece_id=YL7JDBVMKUUHZREXP2OEKCR573NC7RYKFG3BREKQN3Z3LBISJF7A, seg_pos={0 0}, upload=501341]: Done uploading piece. err: <nil> Dec 29 11:19:43.359543432 (571h50m43.230626477s) [noise=true , address=145.79.227.34:4910 , piece_id=OI7WBL7EHIAJTLCMKOLBYJL3ZXBU3AHS7UWJOD3PSPJP3S6V4TBA, seg_pos={0 0}, upload=501341]: Done uploading piece. err: <nil> Dec 29 11:19:43.394210304 (571h50m43.265293349s) [noise=true , address=145.79.227.30:4771 , piece_id=E6UTG5G4RB4F7ARBX4IR3P5NO24CALQ7E3H22KP5OCKZ3VL74ECA, seg_pos={0 0}, upload=501341]: Done uploading piece. err: <nil> Dec 29 11:19:43.448596804 (571h50m43.319679849s) [noise=true , address=45.91.116.10:4094 , piece_id=DMNIOWGGVZFV47SAEPTAQAW7JECUGUQ2JVN7LF4QA23RANX42V3Q, seg_pos={0 0}, upload=501341]: Done uploading piece. err: <nil> Dec 29 11:19:43.448626000 (571h50m43.319709045s) [seg_pos={0 0}, upload=501341]: Segment reached optimal threshold of 14 pieces. Dec 29 11:19:43.448914891 (571h50m43.319997936s) [noise=true , address=77.90.31.11:4588 , piece_id=USOLW6Z6EEZLE2V5XA7N57XBKHQBERW5MXSWPWFWJQMYMWX7MENQ, seg_pos={0 0}, upload=501341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6): protocol: expected piece hash; context canceled; EOF Dec 29 11:19:43.448926743 (571h50m43.320009788s) [noise=true , address=77.90.31.11:4588 , piece_id=USOLW6Z6EEZLE2V5XA7N57XBKHQBERW5MXSWPWFWJQMYMWX7MENQ, seg_pos={0 0}, upload=501341]: Long tail cancellation due to optimal threshold Dec 29 11:19:43.449142253 (571h50m43.320225298s) [noise=true , address=193.149.164.143:4704 , piece_id=XADBOYOBSRNEU6IEN7NCBFRRDGLHV7367V6ARDLENLUDMH76V6SQ, seg_pos={0 0}, upload=501341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 29 11:19:43.449151661 (571h50m43.320234706s) [noise=true , address=193.149.164.143:4704 , piece_id=XADBOYOBSRNEU6IEN7NCBFRRDGLHV7367V6ARDLENLUDMH76V6SQ, seg_pos={0 0}, upload=501341]: Long tail cancellation due to optimal threshold Dec 29 11:19:43.449288411 (571h50m43.320371456s) [noise=true , address=45.91.116.11:4147 , piece_id=GFGWBPMIIFSVET5TIPIX5RHWQ66SQLJS4WAAXV4S75QP7SAZYOBQ, seg_pos={0 0}, upload=501341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YnWetw1NereHQi7cwHCnmu1LDjRFNpB2MQHSWQrjQdN1CZbMw): protocol: expected piece hash; context canceled; EOF Dec 29 11:19:43.449296276 (571h50m43.320379321s) [noise=true , address=45.91.116.11:4147 , piece_id=GFGWBPMIIFSVET5TIPIX5RHWQ66SQLJS4WAAXV4S75QP7SAZYOBQ, seg_pos={0 0}, upload=501341]: Long tail cancellation due to optimal threshold Dec 29 11:19:43.449343737 (571h50m43.320426812s) [noise=true , address=64.49.13.169:4713 , piece_id=G5ETFSQY3WMQH6RWX2XRDHQHDD73VSDUVUEXXBQJHE2RZOSY2L5A, seg_pos={0 0}, upload=501341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:37770->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:37770->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37770->64.49.13.169:4713: use of closed network connection Dec 29 11:19:43.449355399 (571h50m43.320438454s) [noise=true , address=64.49.13.169:4713 , piece_id=G5ETFSQY3WMQH6RWX2XRDHQHDD73VSDUVUEXXBQJHE2RZOSY2L5A, seg_pos={0 0}, upload=501341]: Long tail cancellation due to optimal threshold Dec 29 11:19:43.449382020 (571h50m43.320465075s) [seg_pos={0 0}, upload=501341]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 11:19:43.449485477 (571h50m43.320568522s) [upload=501341]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:19:43.449494904 (571h50m43.32057795s) [upload=501341]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:19:43.738702540 (571h50m43.609785575s) [upload=501341]: Done uploading segments... Dec 29 11:19:43.738721196 (571h50m43.609804241s) [upload=501341]: Upload finished. err: <nil> Duration: 48.244619174s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [..XXX.X..XXX..X.X.XX....X..X.....X.X...X.X.X.X.XX..XX..X..XX] duration: 30.0036484s trace: https://jaeger.dev.storj.io/trace/5d931c9c3b5b7dfb FAIL (2025-12-29 11:21:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 29 11:20:44.023277144 (571h51m43.894360189s) [upload=501359]: Starting upload Dec 29 11:20:44.023339253 (571h51m43.894422298s) [upload=501359]: Done starting upload Dec 29 11:20:44.023370222 (571h51m43.894453267s) [upload=501359]: Uploading segments... Dec 29 11:20:44.023563760 (571h51m43.894646805s) [upload=501359]: Got next segment. Inline: false Dec 29 11:20:44.023595330 (571h51m43.894678365s) [upload=501359]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 11:20:44.023612143 (571h51m43.894695178s) [upload=501359]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 11:20:44.272164835 (571h51m44.14324788s) [seg_pos={0 0}, upload=501359]: Begin upload segment... Dec 29 11:20:44.272211994 (571h51m44.143295039s) [seg_pos={0 0}, upload=501359]: Done begin upload segment. Dec 29 11:20:44.272260317 (571h51m44.143343352s) [noise=true , address=145.79.227.50:4373 , piece_id=D7NWRJRPVUXIBS7WO4O75U7DLFPGG5ZM4JWQGGWTELNCMISW76YQ, seg_pos={0 0}, upload=501359]: Uploading piece... Dec 29 11:20:44.272294001 (571h51m44.143377046s) [noise=true , address=145.79.227.42:4121 , piece_id=OGZSFQDGLLZBJ2UK5D4XB4QTO4MIAC5L43S6G4MIQQS4M5TJNICA, seg_pos={0 0}, upload=501359]: Uploading piece... Dec 29 11:20:44.272939119 (571h51m44.144022164s) [noise=true , address=77.90.31.13:4629 , piece_id=WIXOJEJPDKQK43JS7K7SGOCCB6FMAZ775533BCG65FF3UPINV35Q, seg_pos={0 0}, upload=501359]: Uploading piece... Dec 29 11:20:44.273497151 (571h51m44.144580186s) [noise=true , address=45.91.116.11:4119 , piece_id=G3376GD2VAPMVTHDXPHDMGNBRKZWI35RA5OQHUT7DEUO5VQUYIWA, seg_pos={0 0}, upload=501359]: Uploading piece... Dec 29 11:20:44.273974821 (571h51m44.145057866s) [noise=true , address=45.91.116.5:4300 , piece_id=N5CLDVSCDH4CI2YLUOMVL4S4DJJJYDYQDYL77PBOIMVCJ7YJCYUA, seg_pos={0 0}, upload=501359]: Uploading piece... Dec 29 11:20:44.274491986 (571h51m44.145575021s) [noise=true , address=145.79.227.17:4970 , piece_id=5DVCWW5OTTX6PYCHLHOG4472WZHLNDFAF7MWNBAJD5R6RA4EO45A, seg_pos={0 0}, upload=501359]: Uploading piece... Dec 29 11:20:44.276074459 (571h51m44.147157504s) [noise=true , address=145.79.227.24:4581 , piece_id=67Y6KFZGHA4KPDCFQECXX3LEEZUAY25XRAD4CNK4JYNL2MERCYTA, seg_pos={0 0}, upload=501359]: Uploading piece... Dec 29 11:20:44.276768992 (571h51m44.147852037s) [noise=true , address=145.79.227.14:4838 , piece_id=6CMS4QIRIUKFZYFQNGGUJS3O3ZOBXK3TYGCKOQXB6TZEDR2HTBVQ, seg_pos={0 0}, upload=501359]: Uploading piece... Dec 29 11:20:44.279234446 (571h51m44.150317481s) [noise=true , address=145.79.227.51:4411 , piece_id=YEH73MDDZQWBS6VE7ZS7NRK6SXNOSRDSZMP6XBS4QC6TOCIZLH2A, seg_pos={0 0}, upload=501359]: Uploading piece... Dec 29 11:20:44.280557105 (571h51m44.15164014s) [noise=true , address=145.79.227.36:4517 , piece_id=E7PDBB36OA3MNV262CAWILJBIALEIQIID77Y4VSIDW3CXAENLZWA, seg_pos={0 0}, upload=501359]: Uploading piece... Dec 29 11:20:44.281731281 (571h51m44.152814316s) [noise=true , address=145.79.227.46:4603 , piece_id=N4JZC46KGN2T7USS7ZPZ5RUGPZCYME7O5TKYT6JDOZQZ2SIAK6AQ, seg_pos={0 0}, upload=501359]: Uploading piece... Dec 29 11:20:44.282803383 (571h51m44.153886428s) [noise=true , address=77.90.31.19:4842 , piece_id=XELM2NTPYYLCEOYEBDXC5QMJJQBMLFV2CWLXHT4STLN7FUS3PKCA, seg_pos={0 0}, upload=501359]: Uploading piece... Dec 29 11:20:44.283891244 (571h51m44.154974279s) [noise=true , address=145.79.227.15:4875 , piece_id=WXWEBJTRPL2D5M73QAVQZBH4KF7WE46YUGTJYS4GFHZOO4OL7HMQ, seg_pos={0 0}, upload=501359]: Uploading piece... Dec 29 11:20:44.285385710 (571h51m44.156468785s) [noise=true , address=77.90.31.11:4554 , piece_id=KP6YGATG56AYU3ARPVIC2JLVD2YB7KOP726FKPWIA3ZWQPFEADMQ, seg_pos={0 0}, upload=501359]: Uploading piece... Dec 29 11:20:44.287375288 (571h51m44.158458323s) [noise=true , address=145.79.227.6:4257 , piece_id=MH4LT67DU4N5KH7Y52XPKIMHF2V6EEXDVOGSAQAENVS77KQ3IJBA, seg_pos={0 0}, upload=501359]: Uploading piece... Dec 29 11:20:44.287582573 (571h51m44.158665608s) [noise=true , address=145.79.227.37:4566 , piece_id=CTGJ5VBLVAKBWYVG4W7KXTKD7FVR6CZYG3QKELNWIZXA4TCRS3RA, seg_pos={0 0}, upload=501359]: Uploading piece... Dec 29 11:20:45.976762781 (571h51m45.847845816s) [upload=501359]: Next returned nil segment Dec 29 11:20:45.976782248 (571h51m45.847865283s) [upload=501359]: Waiting for error group managing segments... Dec 29 11:20:46.262970826 (571h51m46.134053871s) [noise=true , address=145.79.227.50:4373 , piece_id=D7NWRJRPVUXIBS7WO4O75U7DLFPGG5ZM4JWQGGWTELNCMISW76YQ, seg_pos={0 0}, upload=501359]: Done uploading piece. err: <nil> Dec 29 11:20:46.263926896 (571h51m46.135009931s) [noise=true , address=145.79.227.14:4838 , piece_id=6CMS4QIRIUKFZYFQNGGUJS3O3ZOBXK3TYGCKOQXB6TZEDR2HTBVQ, seg_pos={0 0}, upload=501359]: Done uploading piece. err: <nil> Dec 29 11:20:46.267844775 (571h51m46.13892781s) [noise=true , address=145.79.227.46:4603 , piece_id=N4JZC46KGN2T7USS7ZPZ5RUGPZCYME7O5TKYT6JDOZQZ2SIAK6AQ, seg_pos={0 0}, upload=501359]: Done uploading piece. err: <nil> Dec 29 11:20:46.268233767 (571h51m46.139316802s) [noise=true , address=145.79.227.17:4970 , piece_id=5DVCWW5OTTX6PYCHLHOG4472WZHLNDFAF7MWNBAJD5R6RA4EO45A, seg_pos={0 0}, upload=501359]: Done uploading piece. err: <nil> Dec 29 11:20:46.270912166 (571h51m46.141995211s) [noise=true , address=145.79.227.37:4566 , piece_id=CTGJ5VBLVAKBWYVG4W7KXTKD7FVR6CZYG3QKELNWIZXA4TCRS3RA, seg_pos={0 0}, upload=501359]: Done uploading piece. err: <nil> Dec 29 11:20:46.271526717 (571h51m46.142609752s) [noise=true , address=145.79.227.15:4875 , piece_id=WXWEBJTRPL2D5M73QAVQZBH4KF7WE46YUGTJYS4GFHZOO4OL7HMQ, seg_pos={0 0}, upload=501359]: Done uploading piece. err: <nil> Dec 29 11:20:46.273000352 (571h51m46.144083388s) [noise=true , address=145.79.227.36:4517 , piece_id=E7PDBB36OA3MNV262CAWILJBIALEIQIID77Y4VSIDW3CXAENLZWA, seg_pos={0 0}, upload=501359]: Done uploading piece. err: <nil> Dec 29 11:20:46.281292052 (571h51m46.152375087s) [noise=true , address=145.79.227.42:4121 , piece_id=OGZSFQDGLLZBJ2UK5D4XB4QTO4MIAC5L43S6G4MIQQS4M5TJNICA, seg_pos={0 0}, upload=501359]: Done uploading piece. err: <nil> Dec 29 11:20:46.281316618 (571h51m46.152399663s) [noise=true , address=145.79.227.6:4257 , piece_id=MH4LT67DU4N5KH7Y52XPKIMHF2V6EEXDVOGSAQAENVS77KQ3IJBA, seg_pos={0 0}, upload=501359]: Done uploading piece. err: <nil> Dec 29 11:20:46.287536452 (571h51m46.158619487s) [noise=true , address=145.79.227.51:4411 , piece_id=YEH73MDDZQWBS6VE7ZS7NRK6SXNOSRDSZMP6XBS4QC6TOCIZLH2A, seg_pos={0 0}, upload=501359]: Done uploading piece. err: <nil> Dec 29 11:20:46.310796082 (571h51m46.181879127s) [noise=true , address=145.79.227.24:4581 , piece_id=67Y6KFZGHA4KPDCFQECXX3LEEZUAY25XRAD4CNK4JYNL2MERCYTA, seg_pos={0 0}, upload=501359]: Done uploading piece. err: <nil> Dec 29 11:20:46.542051648 (571h51m46.413134683s) [noise=true , address=45.91.116.5:4300 , piece_id=N5CLDVSCDH4CI2YLUOMVL4S4DJJJYDYQDYL77PBOIMVCJ7YJCYUA, seg_pos={0 0}, upload=501359]: Done uploading piece. err: <nil> Dec 29 11:20:46.688542680 (571h51m46.559625725s) [noise=true , address=45.91.116.11:4119 , piece_id=G3376GD2VAPMVTHDXPHDMGNBRKZWI35RA5OQHUT7DEUO5VQUYIWA, seg_pos={0 0}, upload=501359]: Done uploading piece. err: <nil> Dec 29 11:20:46.688564221 (571h51m46.559647256s) [seg_pos={0 0}, upload=501359]: Segment reached optimal threshold of 13 pieces. Dec 29 11:20:46.688749293 (571h51m46.559832328s) [noise=true , address=77.90.31.11:4554 , piece_id=KP6YGATG56AYU3ARPVIC2JLVD2YB7KOP726FKPWIA3ZWQPFEADMQ, seg_pos={0 0}, upload=501359]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Dec 29 11:20:46.688756988 (571h51m46.559840033s) [noise=true , address=77.90.31.11:4554 , piece_id=KP6YGATG56AYU3ARPVIC2JLVD2YB7KOP726FKPWIA3ZWQPFEADMQ, seg_pos={0 0}, upload=501359]: Long tail cancellation due to optimal threshold Dec 29 11:20:46.691582417 (571h51m46.562665463s) [noise=true , address=77.90.31.19:4842 , piece_id=XELM2NTPYYLCEOYEBDXC5QMJJQBMLFV2CWLXHT4STLN7FUS3PKCA, seg_pos={0 0}, upload=501359]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 29 11:20:46.691595583 (571h51m46.562678658s) [noise=true , address=77.90.31.19:4842 , piece_id=XELM2NTPYYLCEOYEBDXC5QMJJQBMLFV2CWLXHT4STLN7FUS3PKCA, seg_pos={0 0}, upload=501359]: Long tail cancellation due to optimal threshold Dec 29 11:20:46.691760967 (571h51m46.562844002s) [noise=true , address=77.90.31.13:4629 , piece_id=WIXOJEJPDKQK43JS7K7SGOCCB6FMAZ775533BCG65FF3UPINV35Q, seg_pos={0 0}, upload=501359]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:48250->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:48250->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48250->77.90.31.13:4629: use of closed network connection Dec 29 11:20:46.691768082 (571h51m46.562851127s) [noise=true , address=77.90.31.13:4629 , piece_id=WIXOJEJPDKQK43JS7K7SGOCCB6FMAZ775533BCG65FF3UPINV35Q, seg_pos={0 0}, upload=501359]: Long tail cancellation due to optimal threshold Dec 29 11:20:46.691787959 (571h51m46.562870994s) [seg_pos={0 0}, upload=501359]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 11:20:46.691862381 (571h51m46.562945426s) [upload=501359]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:20:46.691870517 (571h51m46.562953562s) [upload=501359]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:20:46.990935432 (571h51m46.862018477s) [upload=501359]: Done uploading segments... Dec 29 11:20:46.990957735 (571h51m46.86204078s) [upload=501359]: Upload finished. err: <nil> Duration: 30.003663327s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [.XXX......X...X....X...X..........X....X.......X..X...XX..XX] duration: 5.001915447s trace: https://jaeger.dev.storj.io/trace/6c77d78dac3d14a1 FAIL (2025-12-29 11:21:01 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001942408s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [..XXX......X...X....X...X..........X....X.......X..X...XX..X] duration: 5.001432787s trace: https://jaeger.dev.storj.io/trace/1662cd8ebb26d837 FAIL (2025-12-29 11:20:42 UTC) orbiter warning: context deadline exceeded Duration: 5.001453917s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [........X..................................X...............X] duration: 1m0.004314724s trace: https://jaeger.dev.storj.io/trace/30604d8b334c422 FAIL (2025-12-29 11:20:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== s) [noise=true , address=77.90.31.11:4588 , piece_id=USOLW6Z6EEZLE2V5XA7N57XBKHQBERW5MXSWPWFWJQMYMWX7MENQ, seg_pos={0 0}, upload=501341]: Uploading piece... Dec 29 11:19:40.511735860 (571h50m40.382818905s) [noise=true , address=64.49.13.169:4713 , piece_id=G5ETFSQY3WMQH6RWX2XRDHQHDD73VSDUVUEXXBQJHE2RZOSY2L5A, seg_pos={0 0}, upload=501341]: Uploading piece... Dec 29 11:19:40.512126774 (571h50m40.38320982s) [noise=true , address=145.79.227.17:4980 , piece_id=AVU4L6PVWYLJV3D3KZKMG6ZVSYXJC5KVQ5HPUOCJRHBNLRJWA5SA, seg_pos={0 0}, upload=501341]: Uploading piece... Dec 29 11:19:40.512143016 (571h50m40.383226051s) [noise=true , address=145.79.227.34:4910 , piece_id=OI7WBL7EHIAJTLCMKOLBYJL3ZXBU3AHS7UWJOD3PSPJP3S6V4TBA, seg_pos={0 0}, upload=501341]: Uploading piece... Dec 29 11:19:40.512533459 (571h50m40.383616504s) [noise=true , address=145.79.227.43:4178 , piece_id=2BIXT4MN6DT3FTBLWFY5EKN4P5N5RZC2OARQK37WCZE7H4UAAN6A, seg_pos={0 0}, upload=501341]: Uploading piece... Dec 29 11:19:40.512555862 (571h50m40.383638897s) [noise=true , address=145.79.227.44:4205 , piece_id=J5M33FRANHDPN4W4BM23Q6OFQ2REJFVW6DO2LHAKGWJB6GBMTRNQ, seg_pos={0 0}, upload=501341]: Uploading piece... Dec 29 11:19:40.512925265 (571h50m40.38400831s) [noise=true , address=145.79.227.13:4831 , piece_id=YRURTAZFA7NGCRLHTENMVI2KJ6MMIDVFICY32KXDMORS4ZRYJ2AQ, seg_pos={0 0}, upload=501341]: Uploading piece... Dec 29 11:19:40.512947137 (571h50m40.384030172s) [noise=true , address=145.79.227.40:4060 , piece_id=TN5O2G72ZCAYGOUCZDI6JCJOKEUQXWFBE4GOHFXRRX7ENMAYAXZA, seg_pos={0 0}, upload=501341]: Uploading piece... Dec 29 11:19:40.513381343 (571h50m40.384464378s) [noise=true , address=77.90.31.21:4881 , piece_id=U4DNLO5JMNWLRYXPG5QCOMD5M5KM3SHNU53UFSQKLF76E3FVELNA, seg_pos={0 0}, upload=501341]: Uploading piece... Dec 29 11:19:40.513444153 (571h50m40.384527198s) [noise=true , address=145.79.227.30:4771 , piece_id=E6UTG5G4RB4F7ARBX4IR3P5NO24CALQ7E3H22KP5OCKZ3VL74ECA, seg_pos={0 0}, upload=501341]: Uploading piece... Dec 29 11:19:40.513788890 (571h50m40.384871925s) [noise=true , address=45.91.116.11:4147 , piece_id=GFGWBPMIIFSVET5TIPIX5RHWQ66SQLJS4WAAXV4S75QP7SAZYOBQ, seg_pos={0 0}, upload=501341]: Uploading piece... Dec 29 11:19:40.513965336 (571h50m40.385048381s) [noise=true , address=145.79.227.35:4513 , piece_id=XB2F2LAXXP54R3J7R23OOGFMJXAMXDRIWKZTVY2I2OUZG5NBS5PA, seg_pos={0 0}, upload=501341]: Uploading piece... Dec 29 11:19:40.514485947 (571h50m40.385568992s) [noise=true , address=145.79.227.12:4776 , piece_id=4VNNRBRGCHXGTVVU37FH4FYLEIKVIXTVPS4BAFT3EGM62IHGXFEA, seg_pos={0 0}, upload=501341]: Uploading piece... Dec 29 11:19:40.514891830 (571h50m40.385974865s) [noise=true , address=45.91.116.10:4094 , piece_id=DMNIOWGGVZFV47SAEPTAQAW7JECUGUQ2JVN7LF4QA23RANX42V3Q, seg_pos={0 0}, upload=501341]: Uploading piece... Dec 29 11:19:40.515312531 (571h50m40.386395576s) [noise=true , address=145.79.227.33:4903 , piece_id=CZPMPIAJV437JVEPTEJEASZDE2RBWOP7PXEHBPEYBVHD6J2LTFSA, seg_pos={0 0}, upload=501341]: Uploading piece... Dec 29 11:19:40.516926965 (571h50m40.38801001s) [noise=true , address=193.149.164.143:4704 , piece_id=XADBOYOBSRNEU6IEN7NCBFRRDGLHV7367V6ARDLENLUDMH76V6SQ, seg_pos={0 0}, upload=501341]: Uploading piece... Dec 29 11:19:41.844852478 (571h50m41.715935513s) [upload=501341]: Next returned nil segment Dec 29 11:19:41.844866866 (571h50m41.715949901s) [upload=501341]: Waiting for error group managing segments... Dec 29 11:19:42.375579856 (571h50m42.246662891s) [noise=true , address=145.79.227.12:4776 , piece_id=4VNNRBRGCHXGTVVU37FH4FYLEIKVIXTVPS4BAFT3EGM62IHGXFEA, seg_pos={0 0}, upload=501341]: Done uploading piece. err: <nil> Dec 29 11:19:42.376792365 (571h50m42.24787541s) [noise=true , address=145.79.227.17:4980 , piece_id=AVU4L6PVWYLJV3D3KZKMG6ZVSYXJC5KVQ5HPUOCJRHBNLRJWA5SA, seg_pos={0 0}, upload=501341]: Done uploading piece. err: <nil> Dec 29 11:19:42.378045631 (571h50m42.249128676s) [noise=true , address=145.79.227.13:4831 , piece_id=YRURTAZFA7NGCRLHTENMVI2KJ6MMIDVFICY32KXDMORS4ZRYJ2AQ, seg_pos={0 0}, upload=501341]: Done uploading piece. err: <nil> Dec 29 11:19:42.386717985 (571h50m42.25780102s) [noise=true , address=145.79.227.33:4903 , piece_id=CZPMPIAJV437JVEPTEJEASZDE2RBWOP7PXEHBPEYBVHD6J2LTFSA, seg_pos={0 0}, upload=501341]: Done uploading piece. err: <nil> Dec 29 11:19:42.403162488 (571h50m42.274245523s) [noise=true , address=145.79.227.35:4513 , piece_id=XB2F2LAXXP54R3J7R23OOGFMJXAMXDRIWKZTVY2I2OUZG5NBS5PA, seg_pos={0 0}, upload=501341]: Done uploading piece. err: <nil> Dec 29 11:19:42.669549168 (571h50m42.540632214s) [noise=true , address=77.90.31.16:4753 , piece_id=TAS7IRJLJJNFO35X6BTW7UOPSNGVLQEETBUIRI4QFP6ZFKNSN3DA, seg_pos={0 0}, upload=501341]: Done uploading piece. err: <nil> Dec 29 11:19:42.696929825 (571h50m42.56801286s) [noise=true , address=77.90.31.21:4881 , piece_id=U4DNLO5JMNWLRYXPG5QCOMD5M5KM3SHNU53UFSQKLF76E3FVELNA, seg_pos={0 0}, upload=501341]: Done uploading piece. err: <nil> Dec 29 11:19:43.153536604 (571h50m43.024619649s) [noise=true , address=145.79.227.40:4060 , piece_id=TN5O2G72ZCAYGOUCZDI6JCJOKEUQXWFBE4GOHFXRRX7ENMAYAXZA, seg_pos={0 0}, upload=501341]: Done uploading piece. err: <nil> Dec 29 11:19:43.234523837 (571h50m43.105606882s) [noise=true , address=145.79.227.44:4205 , piece_id=J5M33FRANHDPN4W4BM23Q6OFQ2REJFVW6DO2LHAKGWJB6GBMTRNQ, seg_pos={0 0}, upload=501341]: Done uploading piece. err: <nil> Dec 29 11:19:43.306005738 (571h50m43.177088773s) [noise=true , address=145.79.227.43:4178 , piece_id=2BIXT4MN6DT3FTBLWFY5EKN4P5N5RZC2OARQK37WCZE7H4UAAN6A, seg_pos={0 0}, upload=501341]: Done uploading piece. err: <nil> Dec 29 11:19:43.333711425 (571h50m43.20479446s) [noise=true , address=193.149.164.143:4702 , piece_id=YL7JDBVMKUUHZREXP2OEKCR573NC7RYKFG3BREKQN3Z3LBISJF7A, seg_pos={0 0}, upload=501341]: Done uploading piece. err: <nil> Dec 29 11:19:43.359543432 (571h50m43.230626477s) [noise=true , address=145.79.227.34:4910 , piece_id=OI7WBL7EHIAJTLCMKOLBYJL3ZXBU3AHS7UWJOD3PSPJP3S6V4TBA, seg_pos={0 0}, upload=501341]: Done uploading piece. err: <nil> Dec 29 11:19:43.394210304 (571h50m43.265293349s) [noise=true , address=145.79.227.30:4771 , piece_id=E6UTG5G4RB4F7ARBX4IR3P5NO24CALQ7E3H22KP5OCKZ3VL74ECA, seg_pos={0 0}, upload=501341]: Done uploading piece. err: <nil> Dec 29 11:19:43.448596804 (571h50m43.319679849s) [noise=true , address=45.91.116.10:4094 , piece_id=DMNIOWGGVZFV47SAEPTAQAW7JECUGUQ2JVN7LF4QA23RANX42V3Q, seg_pos={0 0}, upload=501341]: Done uploading piece. err: <nil> Dec 29 11:19:43.448626000 (571h50m43.319709045s) [seg_pos={0 0}, upload=501341]: Segment reached optimal threshold of 14 pieces. Dec 29 11:19:43.448914891 (571h50m43.319997936s) [noise=true , address=77.90.31.11:4588 , piece_id=USOLW6Z6EEZLE2V5XA7N57XBKHQBERW5MXSWPWFWJQMYMWX7MENQ, seg_pos={0 0}, upload=501341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6): protocol: expected piece hash; context canceled; EOF Dec 29 11:19:43.448926743 (571h50m43.320009788s) [noise=true , address=77.90.31.11:4588 , piece_id=USOLW6Z6EEZLE2V5XA7N57XBKHQBERW5MXSWPWFWJQMYMWX7MENQ, seg_pos={0 0}, upload=501341]: Long tail cancellation due to optimal threshold Dec 29 11:19:43.449142253 (571h50m43.320225298s) [noise=true , address=193.149.164.143:4704 , piece_id=XADBOYOBSRNEU6IEN7NCBFRRDGLHV7367V6ARDLENLUDMH76V6SQ, seg_pos={0 0}, upload=501341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 29 11:19:43.449151661 (571h50m43.320234706s) [noise=true , address=193.149.164.143:4704 , piece_id=XADBOYOBSRNEU6IEN7NCBFRRDGLHV7367V6ARDLENLUDMH76V6SQ, seg_pos={0 0}, upload=501341]: Long tail cancellation due to optimal threshold Dec 29 11:19:43.449288411 (571h50m43.320371456s) [noise=true , address=45.91.116.11:4147 , piece_id=GFGWBPMIIFSVET5TIPIX5RHWQ66SQLJS4WAAXV4S75QP7SAZYOBQ, seg_pos={0 0}, upload=501341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YnWetw1NereHQi7cwHCnmu1LDjRFNpB2MQHSWQrjQdN1CZbMw): protocol: expected piece hash; context canceled; EOF Dec 29 11:19:43.449296276 (571h50m43.320379321s) [noise=true , address=45.91.116.11:4147 , piece_id=GFGWBPMIIFSVET5TIPIX5RHWQ66SQLJS4WAAXV4S75QP7SAZYOBQ, seg_pos={0 0}, upload=501341]: Long tail cancellation due to optimal threshold Dec 29 11:19:43.449343737 (571h50m43.320426812s) [noise=true , address=64.49.13.169:4713 , piece_id=G5ETFSQY3WMQH6RWX2XRDHQHDD73VSDUVUEXXBQJHE2RZOSY2L5A, seg_pos={0 0}, upload=501341]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:37770->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:37770->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37770->64.49.13.169:4713: use of closed network connection Dec 29 11:19:43.449355399 (571h50m43.320438454s) [noise=true , address=64.49.13.169:4713 , piece_id=G5ETFSQY3WMQH6RWX2XRDHQHDD73VSDUVUEXXBQJHE2RZOSY2L5A, seg_pos={0 0}, upload=501341]: Long tail cancellation due to optimal threshold Dec 29 11:19:43.449382020 (571h50m43.320465075s) [seg_pos={0 0}, upload=501341]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 11:19:43.449485477 (571h50m43.320568522s) [upload=501341]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:19:43.449494904 (571h50m43.32057795s) [upload=501341]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:19:43.738702540 (571h50m43.609785575s) [upload=501341]: Done uploading segments... Dec 29 11:19:43.738721196 (571h50m43.609804241s) [upload=501341]: Upload finished. err: <nil> Duration: 1m0.004333871s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [...XXX.X..XXX..X.X.XX....X..X.....X.X...X.X.X.X.XX..XX..X..X] duration: 30.002971228s trace: https://jaeger.dev.storj.io/trace/6807e2abe25be98a FAIL (2025-12-29 11:20:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: A4KQRBCHN5C46LC7WERZ6ZOVEXXLIJK4Q74W2F25E654VHRLTDUA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: OTVUBDY4SCLKAOK3CGNV65PQPLRCURFNPJQLWBJ5XLRSOCAP75AQ) context deadline exceeded ======LOGS====== Dec 29 11:20:01.726702846 (571h51m1.597785891s) [upload=501350]: Starting upload Dec 29 11:20:01.726799931 (571h51m1.597882976s) [upload=501350]: Done starting upload Dec 29 11:20:01.727132975 (571h51m1.59821602s) [upload=501350]: Uploading segments... Dec 29 11:20:01.727198941 (571h51m1.598281986s) [upload=501350]: Got next segment. Inline: false Dec 29 11:20:01.727241583 (571h51m1.598324668s) [upload=501350]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 11:20:01.727251571 (571h51m1.598334616s) [upload=501350]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 11:20:01.978393804 (571h51m1.849476849s) [seg_pos={0 0}, upload=501350]: Begin upload segment... Dec 29 11:20:01.978466583 (571h51m1.849549628s) [seg_pos={0 0}, upload=501350]: Done begin upload segment. Dec 29 11:20:01.978511409 (571h51m1.849594454s) [noise=true , address=145.79.227.5:4221 , piece_id=YRBTXLUGB3P5REUJ3F677QHOK2LASTMB33AHSYR3Y3FC6QWOOR3A, seg_pos={0 0}, upload=501350]: Uploading piece... Dec 29 11:20:01.978759441 (571h51m1.849842496s) [noise=true , address=145.79.227.10:4401 , piece_id=JQIGQXFBOSUMX7GG5WDFFWMC6ICUARD4R2FHECRI3ULI5KWRBDTQ, seg_pos={0 0}, upload=501350]: Uploading piece... Dec 29 11:20:01.979401643 (571h51m1.850484688s) [noise=true , address=145.79.227.50:4374 , piece_id=BPW3DGRNVLJZO4DQZMCJREDT37BGNALFUYINMIUBKFLIVS2JIKGA, seg_pos={0 0}, upload=501350]: Uploading piece... Dec 29 11:20:01.979918558 (571h51m1.851001633s) [noise=true , address=145.79.227.14:4838 , piece_id=IEIMIWRPM7LORZSZV5WDA5JHQXSXNFVVWVLNUIY2OX6V5YWMKCQQ, seg_pos={0 0}, upload=501350]: Uploading piece... Dec 29 11:20:01.980359488 (571h51m1.851442533s) [noise=true , address=45.91.116.5:4300 , piece_id=ANGO4ISEUIOS32HHFUU5UGVEYABCKR2LLT7XDRUXR747LT4PC34A, seg_pos={0 0}, upload=501350]: Uploading piece... Dec 29 11:20:01.980866983 (571h51m1.851950028s) [noise=true , address=145.79.227.42:4122 , piece_id=MBCYFXJW4QYSJAUVQWYQQATK63AVPVMB25AWG2JNSLHBHVWPTLNQ, seg_pos={0 0}, upload=501350]: Uploading piece... Dec 29 11:20:01.981443962 (571h51m1.852527008s) [noise=true , address=145.79.227.6:4259 , piece_id=E7WOCZ4MXJBPJSDTZQ35RLI2AIH6UWBZWQWLI5Z5MW46SGNFYZZA, seg_pos={0 0}, upload=501350]: Uploading piece... Dec 29 11:20:01.981983239 (571h51m1.853066294s) [noise=true , address=145.79.227.41:4087 , piece_id=5Q636L5RK356JL5OP6AUAUBBVMONEQMOC7VWUDHB6HDI36P7NWAA, seg_pos={0 0}, upload=501350]: Uploading piece... Dec 29 11:20:01.982320622 (571h51m1.853403667s) [noise=true , address=145.79.227.15:4876 , piece_id=RVHN7WW5QO3LLRCT6DXX7NEMRAXYGC5QGGWCHKY72P5PFFK4JJJQ, seg_pos={0 0}, upload=501350]: Uploading piece... Dec 29 11:20:01.982526524 (571h51m1.853609569s) [noise=true , address=145.79.227.44:4194 , piece_id=UN7FST7L6VABMOST6JZZWHMD5QBKUEDKGAVALJ53Z7DPQWR2436A, seg_pos={0 0}, upload=501350]: Uploading piece... Dec 29 11:20:01.982934670 (571h51m1.854017725s) [noise=true , address=77.90.31.10:4518 , piece_id=A4KQRBCHN5C46LC7WERZ6ZOVEXXLIJK4Q74W2F25E654VHRLTDUA, seg_pos={0 0}, upload=501350]: Uploading piece... Dec 29 11:20:01.983424833 (571h51m1.854507879s) [noise=true , address=45.91.116.9:4261 , piece_id=OTVUBDY4SCLKAOK3CGNV65PQPLRCURFNPJQLWBJ5XLRSOCAP75AQ, seg_pos={0 0}, upload=501350]: Uploading piece... Dec 29 11:20:01.983870753 (571h51m1.854953798s) [noise=true , address=145.79.227.47:4265 , piece_id=IN2PCZQT7EXI2OCDMT7PYOW7VBGMT6G32P26GYQB3OM2LFPIX2WA, seg_pos={0 0}, upload=501350]: Uploading piece... Dec 29 11:20:01.984549556 (571h51m1.855632601s) [noise=true , address=77.90.31.21:4877 , piece_id=PI2EPVCSWSFDQX3MPB5LOJNHQ444O2JG6C37V63ZJMO4V5EGJDOA, seg_pos={0 0}, upload=501350]: Uploading piece... Dec 29 11:20:01.986932513 (571h51m1.858015598s) [noise=true , address=145.79.227.52:4446 , piece_id=2XICIYILLIBJMYRZ44AIUODS6UOTHITSDJWULVLGVL7QCBGFDRGQ, seg_pos={0 0}, upload=501350]: Uploading piece... Dec 29 11:20:01.987726505 (571h51m1.85880955s) [noise=true , address=145.79.227.32:4834 , piece_id=HB4NIJPO6ZNIYYSGM7NR4KEWTAW3ISJVTAELLFP2NPB7I7M4PXVA, seg_pos={0 0}, upload=501350]: Uploading piece... Dec 29 11:20:03.634297598 (571h51m3.505380643s) [upload=501350]: Next returned nil segment Dec 29 11:20:03.634319800 (571h51m3.505402845s) [upload=501350]: Waiting for error group managing segments... Dec 29 11:20:04.031775466 (571h51m3.902858511s) [noise=true , address=145.79.227.15:4876 , piece_id=RVHN7WW5QO3LLRCT6DXX7NEMRAXYGC5QGGWCHKY72P5PFFK4JJJQ, seg_pos={0 0}, upload=501350]: Done uploading piece. err: <nil> Dec 29 11:20:04.050606813 (571h51m3.921689859s) [noise=true , address=145.79.227.32:4834 , piece_id=HB4NIJPO6ZNIYYSGM7NR4KEWTAW3ISJVTAELLFP2NPB7I7M4PXVA, seg_pos={0 0}, upload=501350]: Done uploading piece. err: <nil> Dec 29 11:20:04.050797597 (571h51m3.921880652s) [noise=true , address=145.79.227.10:4401 , piece_id=JQIGQXFBOSUMX7GG5WDFFWMC6ICUARD4R2FHECRI3ULI5KWRBDTQ, seg_pos={0 0}, upload=501350]: Done uploading piece. err: <nil> Dec 29 11:20:04.051464697 (571h51m3.922547742s) [noise=true , address=145.79.227.50:4374 , piece_id=BPW3DGRNVLJZO4DQZMCJREDT37BGNALFUYINMIUBKFLIVS2JIKGA, seg_pos={0 0}, upload=501350]: Done uploading piece. err: <nil> Dec 29 11:20:04.051781981 (571h51m3.922865036s) [noise=true , address=145.79.227.41:4087 , piece_id=5Q636L5RK356JL5OP6AUAUBBVMONEQMOC7VWUDHB6HDI36P7NWAA, seg_pos={0 0}, upload=501350]: Done uploading piece. err: <nil> Dec 29 11:20:04.052181762 (571h51m3.923264817s) [noise=true , address=145.79.227.42:4122 , piece_id=MBCYFXJW4QYSJAUVQWYQQATK63AVPVMB25AWG2JNSLHBHVWPTLNQ, seg_pos={0 0}, upload=501350]: Done uploading piece. err: <nil> Dec 29 11:20:04.053649236 (571h51m3.924732281s) [noise=true , address=145.79.227.6:4259 , piece_id=E7WOCZ4MXJBPJSDTZQ35RLI2AIH6UWBZWQWLI5Z5MW46SGNFYZZA, seg_pos={0 0}, upload=501350]: Done uploading piece. err: <nil> Dec 29 11:20:04.053768344 (571h51m3.924851389s) [noise=true , address=145.79.227.52:4446 , piece_id=2XICIYILLIBJMYRZ44AIUODS6UOTHITSDJWULVLGVL7QCBGFDRGQ, seg_pos={0 0}, upload=501350]: Done uploading piece. err: <nil> Dec 29 11:20:04.054760753 (571h51m3.925843808s) [noise=true , address=145.79.227.14:4838 , piece_id=IEIMIWRPM7LORZSZV5WDA5JHQXSXNFVVWVLNUIY2OX6V5YWMKCQQ, seg_pos={0 0}, upload=501350]: Done uploading piece. err: <nil> Dec 29 11:20:04.055167147 (571h51m3.926250192s) [noise=true , address=145.79.227.47:4265 , piece_id=IN2PCZQT7EXI2OCDMT7PYOW7VBGMT6G32P26GYQB3OM2LFPIX2WA, seg_pos={0 0}, upload=501350]: Done uploading piece. err: <nil> Dec 29 11:20:04.056715866 (571h51m3.927798911s) [noise=true , address=145.79.227.5:4221 , piece_id=YRBTXLUGB3P5REUJ3F677QHOK2LASTMB33AHSYR3Y3FC6QWOOR3A, seg_pos={0 0}, upload=501350]: Done uploading piece. err: <nil> Dec 29 11:20:04.382130615 (571h51m4.25321366s) [noise=true , address=45.91.116.9:4261 , piece_id=OTVUBDY4SCLKAOK3CGNV65PQPLRCURFNPJQLWBJ5XLRSOCAP75AQ, seg_pos={0 0}, upload=501350]: Done uploading piece. err: <nil> Dec 29 11:20:04.383379944 (571h51m4.25446299s) [noise=true , address=77.90.31.10:4518 , piece_id=A4KQRBCHN5C46LC7WERZ6ZOVEXXLIJK4Q74W2F25E654VHRLTDUA, seg_pos={0 0}, upload=501350]: Done uploading piece. err: <nil> Dec 29 11:20:04.383436472 (571h51m4.254519528s) [seg_pos={0 0}, upload=501350]: Segment reached optimal threshold of 13 pieces. Dec 29 11:20:04.383869416 (571h51m4.254952461s) [noise=true , address=77.90.31.21:4877 , piece_id=PI2EPVCSWSFDQX3MPB5LOJNHQ444O2JG6C37V63ZJMO4V5EGJDOA, seg_pos={0 0}, upload=501350]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 29 11:20:04.383882843 (571h51m4.254965898s) [noise=true , address=77.90.31.21:4877 , piece_id=PI2EPVCSWSFDQX3MPB5LOJNHQ444O2JG6C37V63ZJMO4V5EGJDOA, seg_pos={0 0}, upload=501350]: Long tail cancellation due to optimal threshold Dec 29 11:20:04.384253357 (571h51m4.255336423s) [noise=true , address=45.91.116.5:4300 , piece_id=ANGO4ISEUIOS32HHFUU5UGVEYABCKR2LLT7XDRUXR747LT4PC34A, seg_pos={0 0}, upload=501350]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 29 11:20:04.384265712 (571h51m4.255348767s) [noise=true , address=45.91.116.5:4300 , piece_id=ANGO4ISEUIOS32HHFUU5UGVEYABCKR2LLT7XDRUXR747LT4PC34A, seg_pos={0 0}, upload=501350]: Long tail cancellation due to optimal threshold Dec 29 11:20:04.384590630 (571h51m4.255673675s) [noise=true , address=145.79.227.44:4194 , piece_id=UN7FST7L6VABMOST6JZZWHMD5QBKUEDKGAVALJ53Z7DPQWR2436A, seg_pos={0 0}, upload=501350]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 29 11:20:04.384603544 (571h51m4.255686589s) [noise=true , address=145.79.227.44:4194 , piece_id=UN7FST7L6VABMOST6JZZWHMD5QBKUEDKGAVALJ53Z7DPQWR2436A, seg_pos={0 0}, upload=501350]: Long tail cancellation due to optimal threshold Dec 29 11:20:04.384634153 (571h51m4.255717198s) [seg_pos={0 0}, upload=501350]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 11:20:04.384874901 (571h51m4.255957946s) [upload=501350]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:20:04.384882235 (571h51m4.25596527s) [upload=501350]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:20:04.686253251 (571h51m4.557336296s) [upload=501350]: Done uploading segments... Dec 29 11:20:04.686327061 (571h51m4.557410106s) [upload=501350]: Upload finished. err: <nil> Duration: 30.003013959s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [...XXX......X...X....X...X..........X....X.......X..X...XX..] duration: 5.000373438s trace: https://jaeger.dev.storj.io/trace/73a9bf4afa47bf95 FAIL (2025-12-29 11:19:44 UTC) orbiter warning: context deadline exceeded Duration: 5.000394489s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [....XXX.X..XXX..X.X.XX....X..X.....X.X...X.X.X.X.XX..XX..X..] duration: 6.006445453s trace: https://jaeger.dev.storj.io/trace/75fd8864c326a461 FAIL (2025-12-29 11:19:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 99200282 (571h48m50.970283317s) [upload=501315]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 11:17:51.346179789 (571h48m51.217262824s) [seg_pos={0 0}, upload=501315]: Begin upload segment... Dec 29 11:17:51.346222831 (571h48m51.217305866s) [seg_pos={0 0}, upload=501315]: Done begin upload segment. Dec 29 11:17:51.346257217 (571h48m51.217340262s) [noise=true , address=145.79.227.6:4258 , piece_id=APORUCBLL7PMK24T67F5BTCQC4XI34F2DK76ONV4LL5BSZQTEKNA, seg_pos={0 0}, upload=501315]: Uploading piece... Dec 29 11:17:51.346382686 (571h48m51.217465731s) [noise=true , address=77.90.31.21:4878 , piece_id=ZT5F6UBC7AIIVQ6IKESR2QL2USYAXEJJVRZW32FQQJW7FME74NGA, seg_pos={0 0}, upload=501315]: Uploading piece... Dec 29 11:17:51.346906814 (571h48m51.217989849s) [noise=true , address=145.79.227.41:4086 , piece_id=2SYW2ODOYJLMLDBPMVMWX666DGSI3EP3DYIGS6PQUUKRCIN6WHPA, seg_pos={0 0}, upload=501315]: Uploading piece... Dec 29 11:17:51.347609662 (571h48m51.218692707s) [noise=true , address=77.90.31.10:4517 , piece_id=A5OQB4OQ7OFF3TTX6YOWCW7XI5UZVUJQZA46RZA6JKUCJOSUIG7A, seg_pos={0 0}, upload=501315]: Uploading piece... Dec 29 11:17:51.348249380 (571h48m51.219332425s) [noise=true , address=145.79.227.4:4185 , piece_id=MMQDNKVGRHKANA5RJFEAQGK7TTYDQGB36F7G3Y4ZMVGJ4QGOFAZQ, seg_pos={0 0}, upload=501315]: Uploading piece... Dec 29 11:17:51.348779831 (571h48m51.219862866s) [noise=true , address=145.79.227.16:4909 , piece_id=YVHLZE2PVEZMGKPWEJCHG327TLMEUREGB3HG6WM6DOE5LMNTZDOA, seg_pos={0 0}, upload=501315]: Uploading piece... Dec 29 11:17:51.349326631 (571h48m51.220409676s) [noise=true , address=77.90.31.14:4663 , piece_id=CJ2PL6JR5QVGWSWQZ2LKEWYLU6JSZFCHTFGBJDRW36MOHCHI74KA, seg_pos={0 0}, upload=501315]: Uploading piece... Dec 29 11:17:51.350979048 (571h48m51.222062083s) [noise=true , address=145.79.227.39:4013 , piece_id=CNIJUUM3G5OXCK2YDBMRVVZWYNQ46XWDTMTNOTX242PQ2MJDUU7Q, seg_pos={0 0}, upload=501315]: Uploading piece... Dec 29 11:17:51.351737461 (571h48m51.222820507s) [noise=true , address=145.79.227.15:4875 , piece_id=DJQNISDBMSV5WCCHH2F7BIGPVU2RGCOYM6WUVYXRMEA6H5PIRDCQ, seg_pos={0 0}, upload=501315]: Uploading piece... Dec 29 11:17:51.352905997 (571h48m51.223989032s) [noise=true , address=145.79.227.37:4567 , piece_id=MHPYTO4SGHNGUPE5I7PK55CCCOTPX7MJV6NQE2W77I7FETICCVYQ, seg_pos={0 0}, upload=501315]: Uploading piece... Dec 29 11:17:51.354573763 (571h48m51.225656808s) [noise=true , address=45.91.116.12:4226 , piece_id=PRLMBIXK3HBGKH32A36FOCWZNPBH4LUQUT3Y6JWNMHEQDKIZQXPQ, seg_pos={0 0}, upload=501315]: Uploading piece... Dec 29 11:17:51.362073954 (571h48m51.233156999s) [noise=true , address=145.79.227.47:4267 , piece_id=NWLAQGZKUIG2HXQVZ3ALADBQYGTXH5E4A54NJSLUGP7P637ZOBOQ, seg_pos={0 0}, upload=501315]: Uploading piece... Dec 29 11:17:51.362708994 (571h48m51.233792039s) [noise=true , address=145.79.227.28:4726 , piece_id=SKWAZQIVJMBFX4E7O3C4BLL7TIIBRO3DIWQ2O355CDFVHQCOGKVA, seg_pos={0 0}, upload=501315]: Uploading piece... Dec 29 11:17:51.363354202 (571h48m51.234437237s) [noise=true , address=77.90.31.18:4808 , piece_id=AEG3RKBXSMEZZUUASSG76ZCWG6X2G6TM7PQLNFEO4QONMAD2L6XA, seg_pos={0 0}, upload=501315]: Uploading piece... Dec 29 11:17:51.365273356 (571h48m51.236356401s) [noise=true , address=145.79.227.36:4519 , piece_id=DINWMLIK55GANRIUXO5ZPTBKUSDWDBFC77CGEPYUZW52LOPVFGSA, seg_pos={0 0}, upload=501315]: Uploading piece... Dec 29 11:17:51.366491546 (571h48m51.237574591s) [noise=true , address=45.91.116.13:4153 , piece_id=TFFVKLM6CH5EV423HR6J3MTFKSAVS5CEYC7W5GXC2HXZYQSAUHEA, seg_pos={0 0}, upload=501315]: Uploading piece... Dec 29 11:17:53.052290295 (571h48m52.92337334s) [upload=501315]: Next returned nil segment Dec 29 11:17:53.052302259 (571h48m52.923385304s) [upload=501315]: Waiting for error group managing segments... Dec 29 11:17:53.336558997 (571h48m53.207642032s) [noise=true , address=145.79.227.4:4185 , piece_id=MMQDNKVGRHKANA5RJFEAQGK7TTYDQGB36F7G3Y4ZMVGJ4QGOFAZQ, seg_pos={0 0}, upload=501315]: Done uploading piece. err: <nil> Dec 29 11:17:53.338719751 (571h48m53.209802786s) [noise=true , address=145.79.227.41:4086 , piece_id=2SYW2ODOYJLMLDBPMVMWX666DGSI3EP3DYIGS6PQUUKRCIN6WHPA, seg_pos={0 0}, upload=501315]: Done uploading piece. err: <nil> Dec 29 11:17:53.348516166 (571h48m53.219599211s) [noise=true , address=145.79.227.36:4519 , piece_id=DINWMLIK55GANRIUXO5ZPTBKUSDWDBFC77CGEPYUZW52LOPVFGSA, seg_pos={0 0}, upload=501315]: Done uploading piece. err: <nil> Dec 29 11:17:53.349092463 (571h48m53.220175498s) [noise=true , address=145.79.227.6:4258 , piece_id=APORUCBLL7PMK24T67F5BTCQC4XI34F2DK76ONV4LL5BSZQTEKNA, seg_pos={0 0}, upload=501315]: Done uploading piece. err: <nil> Dec 29 11:17:53.351559340 (571h48m53.222642385s) [noise=true , address=145.79.227.37:4567 , piece_id=MHPYTO4SGHNGUPE5I7PK55CCCOTPX7MJV6NQE2W77I7FETICCVYQ, seg_pos={0 0}, upload=501315]: Done uploading piece. err: <nil> Dec 29 11:17:53.354796334 (571h48m53.225879369s) [noise=true , address=145.79.227.28:4726 , piece_id=SKWAZQIVJMBFX4E7O3C4BLL7TIIBRO3DIWQ2O355CDFVHQCOGKVA, seg_pos={0 0}, upload=501315]: Done uploading piece. err: <nil> Dec 29 11:17:53.359184780 (571h48m53.230267815s) [noise=true , address=145.79.227.39:4013 , piece_id=CNIJUUM3G5OXCK2YDBMRVVZWYNQ46XWDTMTNOTX242PQ2MJDUU7Q, seg_pos={0 0}, upload=501315]: Done uploading piece. err: <nil> Dec 29 11:17:53.366321790 (571h48m53.237404825s) [noise=true , address=145.79.227.16:4909 , piece_id=YVHLZE2PVEZMGKPWEJCHG327TLMEUREGB3HG6WM6DOE5LMNTZDOA, seg_pos={0 0}, upload=501315]: Done uploading piece. err: <nil> Dec 29 11:17:53.371482737 (571h48m53.242565783s) [noise=true , address=145.79.227.15:4875 , piece_id=DJQNISDBMSV5WCCHH2F7BIGPVU2RGCOYM6WUVYXRMEA6H5PIRDCQ, seg_pos={0 0}, upload=501315]: Done uploading piece. err: <nil> Dec 29 11:17:53.395127551 (571h48m53.266210596s) [noise=true , address=145.79.227.47:4267 , piece_id=NWLAQGZKUIG2HXQVZ3ALADBQYGTXH5E4A54NJSLUGP7P637ZOBOQ, seg_pos={0 0}, upload=501315]: Done uploading piece. err: <nil> Dec 29 11:17:53.556286081 (571h48m53.427369126s) [noise=true , address=77.90.31.21:4878 , piece_id=ZT5F6UBC7AIIVQ6IKESR2QL2USYAXEJJVRZW32FQQJW7FME74NGA, seg_pos={0 0}, upload=501315]: Done uploading piece. err: <nil> Dec 29 11:17:53.759919451 (571h48m53.631002486s) [noise=true , address=45.91.116.13:4153 , piece_id=TFFVKLM6CH5EV423HR6J3MTFKSAVS5CEYC7W5GXC2HXZYQSAUHEA, seg_pos={0 0}, upload=501315]: Done uploading piece. err: <nil> Dec 29 11:17:53.885088050 (571h48m53.756171096s) [noise=true , address=77.90.31.18:4808 , piece_id=AEG3RKBXSMEZZUUASSG76ZCWG6X2G6TM7PQLNFEO4QONMAD2L6XA, seg_pos={0 0}, upload=501315]: Done uploading piece. err: <nil> Dec 29 11:17:53.885110774 (571h48m53.756193809s) [seg_pos={0 0}, upload=501315]: Segment reached optimal threshold of 13 pieces. Dec 29 11:17:53.885424251 (571h48m53.756507286s) [noise=true , address=45.91.116.12:4226 , piece_id=PRLMBIXK3HBGKH32A36FOCWZNPBH4LUQUT3Y6JWNMHEQDKIZQXPQ, seg_pos={0 0}, upload=501315]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: write tcp 188.166.225.170:44604->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:44604->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44604->45.91.116.12:4226: use of closed network connection Dec 29 11:17:53.885432537 (571h48m53.756515572s) [noise=true , address=45.91.116.12:4226 , piece_id=PRLMBIXK3HBGKH32A36FOCWZNPBH4LUQUT3Y6JWNMHEQDKIZQXPQ, seg_pos={0 0}, upload=501315]: Long tail cancellation due to optimal threshold Dec 29 11:17:53.885731496 (571h48m53.756814551s) [noise=true , address=77.90.31.14:4663 , piece_id=CJ2PL6JR5QVGWSWQZ2LKEWYLU6JSZFCHTFGBJDRW36MOHCHI74KA, seg_pos={0 0}, upload=501315]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: write tcp 188.166.225.170:42844->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:42844->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42844->77.90.31.14:4663: use of closed network connection Dec 29 11:17:53.885741044 (571h48m53.756824089s) [noise=true , address=77.90.31.14:4663 , piece_id=CJ2PL6JR5QVGWSWQZ2LKEWYLU6JSZFCHTFGBJDRW36MOHCHI74KA, seg_pos={0 0}, upload=501315]: Long tail cancellation due to optimal threshold Dec 29 11:17:53.886166634 (571h48m53.757249679s) [noise=true , address=77.90.31.10:4517 , piece_id=A5OQB4OQ7OFF3TTX6YOWCW7XI5UZVUJQZA46RZA6JKUCJOSUIG7A, seg_pos={0 0}, upload=501315]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: write tcp 188.166.225.170:48834->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:48834->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48834->77.90.31.10:4517: use of closed network connection Dec 29 11:17:53.886186873 (571h48m53.757269918s) [noise=true , address=77.90.31.10:4517 , piece_id=A5OQB4OQ7OFF3TTX6YOWCW7XI5UZVUJQZA46RZA6JKUCJOSUIG7A, seg_pos={0 0}, upload=501315]: Long tail cancellation due to optimal threshold Dec 29 11:17:53.886271935 (571h48m53.75735497s) [seg_pos={0 0}, upload=501315]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 11:17:53.886413795 (571h48m53.75749684s) [upload=501315]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:17:53.886420729 (571h48m53.757503764s) [upload=501315]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:17:54.180885525 (571h48m54.05196856s) [upload=501315]: Done uploading segments... Dec 29 11:17:54.180904801 (571h48m54.051987846s) [upload=501315]: Upload finished. err: <nil> Duration:* 6.006471553s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X..................................X...............] duration: 22.846472243s trace: https://jaeger.dev.storj.io/trace/2a822012276bcc5d FAIL (2025-12-29 11:18:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== oise=true , address=45.91.116.12:4239 , piece_id=5JWXVPMXPTWP53E6PIJULVD55R5S46DQ6KR2ZGGQATDQGZFG4LVA, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.882470730 (571h36m32.753553775s) [noise=true , address=77.90.31.10:4527 , piece_id=MRJAMMWFYNEXW32PA654KUVODUH3E5F2RLZQSUJTU55DARPGOLTQ, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.882771553 (571h36m32.753854588s) [noise=true , address=77.90.31.13:4660 , piece_id=WOBMTE3FTJZ6XGZBFWQE4GD6WIWQOAYO4QQMV75YA3O2SV2Y7S3A, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.883107963 (571h36m32.754190998s) [noise=true , address=77.90.31.19:4870 , piece_id=52VQK332LJWMGH4IVSANBAF7LA3JSTVS6CRSS3QCG3UNU7IOOHIA, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.883533073 (571h36m32.754616118s) [noise=true , address=77.90.31.21:4884 , piece_id=75FG6IG4OGITIQXYBG4CPSBA32M5EI6BEA4GG4ULTNVXUWEVY3FA, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.883944547 (571h36m32.755027582s) [noise=true , address=145.79.227.49:4355 , piece_id=4E2QHCBUJB7EHCSNBT3CHMTWVZZ3XEZ5AQ6ZI5Q5EZSWLV6B6LUQ, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.884326042 (571h36m32.755409077s) [noise=true , address=145.79.227.40:4069 , piece_id=IGIQ6YEI2FZJMPGG7APVRI4FE5SWBFLSEMUKVLBJ4VJNCEY7I7VQ, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.886067960 (571h36m32.757150995s) [noise=true , address=77.90.31.11:4570 , piece_id=YUGNQGA2ZEAR3ALX4K35IOEO5KLG7HKG5GKLY3QW5RFYCDRW64QQ, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.886589914 (571h36m32.757672949s) [noise=true , address=145.79.227.34:4930 , piece_id=3S54QR4S6E3J6J5NM2WW3WBVO6W2TPBZZ5ZPH46CSJOA5PA5ER2A, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.886895325 (571h36m32.75797837s) [noise=true , address=64.49.13.140:4516 , piece_id=MS324APTVZJZW3LS4A2ASDK4TASDSHNPHPZTJTPBOCX7NQHSI2CA, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.888604990 (571h36m32.759688035s) [noise=true , address=145.79.227.44:4218 , piece_id=RQSCIRURKNQ24BANRRMMO6WWDSAHGBPABXT7U72ZGGI5HZY5KXQQ, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.888877790 (571h36m32.759960925s) [noise=true , address=64.49.13.169:4707 , piece_id=NL2EYMTZOVRQQ4APNVBGE66HJA5QILLU6TFB5TDBGBYU3GO4IRHA, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.890542230 (571h36m32.761625275s) [noise=true , address=64.49.13.139:4469 , piece_id=QBJJPHHVJPXZSE7V3JK6QLYCSLGX2JCFKR7U4BTEVDK6I5PCBY5Q, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.893184912 (571h36m32.764267947s) [noise=true , address=145.79.227.5:4241 , piece_id=JJIYRPV4Z5LGQNV6YL32OTNSZUG3QBSY47S2RM3RU2LU52NMBWZQ, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.896004932 (571h36m32.767087967s) [noise=true , address=145.79.227.33:4885 , piece_id=NDUWA52AR3E5Z5B47W7TQQYYP5JNOSVY3IVOHB5AIK27NOG4DBRQ, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:32.901393461 (571h36m32.772476507s) [noise=true , address=145.79.227.49:4370 , piece_id=FQZZMWFSFDE55SRUZQSOZXRNX5A3BOQK4H2GH2BRWRABKXG5OZOQ, seg_pos={0 0}, upload=501136]: Uploading piece... Dec 29 11:05:34.200413331 (571h36m34.071496366s) [upload=501136]: Next returned nil segment Dec 29 11:05:34.200437376 (571h36m34.071520411s) [upload=501136]: Waiting for error group managing segments... Dec 29 11:05:34.531472608 (571h36m34.402555643s) [noise=true , address=64.49.13.139:4469 , piece_id=QBJJPHHVJPXZSE7V3JK6QLYCSLGX2JCFKR7U4BTEVDK6I5PCBY5Q, seg_pos={0 0}, upload=501136]: Done uploading piece. err: <nil> Dec 29 11:05:34.589291783 (571h36m34.460374819s) [noise=true , address=64.49.13.140:4516 , piece_id=MS324APTVZJZW3LS4A2ASDK4TASDSHNPHPZTJTPBOCX7NQHSI2CA, seg_pos={0 0}, upload=501136]: Done uploading piece. err: <nil> Dec 29 11:05:34.782400973 (571h36m34.653484008s) [noise=true , address=145.79.227.33:4885 , piece_id=NDUWA52AR3E5Z5B47W7TQQYYP5JNOSVY3IVOHB5AIK27NOG4DBRQ, seg_pos={0 0}, upload=501136]: Done uploading piece. err: <nil> Dec 29 11:05:34.895054052 (571h36m34.766137097s) [noise=true , address=145.79.227.34:4930 , piece_id=3S54QR4S6E3J6J5NM2WW3WBVO6W2TPBZZ5ZPH46CSJOA5PA5ER2A, seg_pos={0 0}, upload=501136]: Done uploading piece. err: <nil> Dec 29 11:05:34.965988150 (571h36m34.837071195s) [noise=true , address=145.79.227.5:4241 , piece_id=JJIYRPV4Z5LGQNV6YL32OTNSZUG3QBSY47S2RM3RU2LU52NMBWZQ, seg_pos={0 0}, upload=501136]: Done uploading piece. err: <nil> Dec 29 11:05:35.068876977 (571h36m34.939960012s) [noise=true , address=77.90.31.19:4870 , piece_id=52VQK332LJWMGH4IVSANBAF7LA3JSTVS6CRSS3QCG3UNU7IOOHIA, seg_pos={0 0}, upload=501136]: Done uploading piece. err: <nil> Dec 29 11:05:35.075857198 (571h36m34.946940243s) [noise=true , address=77.90.31.11:4570 , piece_id=YUGNQGA2ZEAR3ALX4K35IOEO5KLG7HKG5GKLY3QW5RFYCDRW64QQ, seg_pos={0 0}, upload=501136]: Done uploading piece. err: <nil> Dec 29 11:05:35.237630089 (571h36m35.108713134s) [noise=true , address=77.90.31.13:4660 , piece_id=WOBMTE3FTJZ6XGZBFWQE4GD6WIWQOAYO4QQMV75YA3O2SV2Y7S3A, seg_pos={0 0}, upload=501136]: Done uploading piece. err: <nil> Dec 29 11:05:35.371788215 (571h36m35.24287126s) [noise=true , address=77.90.31.21:4884 , piece_id=75FG6IG4OGITIQXYBG4CPSBA32M5EI6BEA4GG4ULTNVXUWEVY3FA, seg_pos={0 0}, upload=501136]: Done uploading piece. err: <nil> Dec 29 11:05:35.635729582 (571h36m35.506812627s) [noise=true , address=145.79.227.30:4784 , piece_id=BUNTKFESZUDAYXSOIMKUG7DVUF65CQJAHDB7NVHII4A5LEWA4K4Q, seg_pos={0 0}, upload=501136]: Done uploading piece. err: <nil> Dec 29 11:05:35.650702903 (571h36m35.521785948s) [noise=true , address=145.79.227.44:4218 , piece_id=RQSCIRURKNQ24BANRRMMO6WWDSAHGBPABXT7U72ZGGI5HZY5KXQQ, seg_pos={0 0}, upload=501136]: Done uploading piece. err: <nil> Dec 29 11:05:35.721297946 (571h36m35.592380991s) [noise=true , address=145.79.227.49:4355 , piece_id=4E2QHCBUJB7EHCSNBT3CHMTWVZZ3XEZ5AQ6ZI5Q5EZSWLV6B6LUQ, seg_pos={0 0}, upload=501136]: Done uploading piece. err: <nil> Dec 29 11:05:35.812319244 (571h36m35.683402279s) [noise=true , address=77.90.31.10:4527 , piece_id=MRJAMMWFYNEXW32PA654KUVODUH3E5F2RLZQSUJTU55DARPGOLTQ, seg_pos={0 0}, upload=501136]: Done uploading piece. err: <nil> Dec 29 11:05:36.101131227 (571h36m35.972214262s) [noise=true , address=145.79.227.49:4370 , piece_id=FQZZMWFSFDE55SRUZQSOZXRNX5A3BOQK4H2GH2BRWRABKXG5OZOQ, seg_pos={0 0}, upload=501136]: Done uploading piece. err: <nil> Dec 29 11:05:36.101172385 (571h36m35.97225542s) [seg_pos={0 0}, upload=501136]: Segment reached optimal threshold of 14 pieces. Dec 29 11:05:36.101312391 (571h36m35.972395436s) [noise=true , address=45.91.116.5:4318 , piece_id=PWFEBU4DLSDME6FHXGHE55U6Y7KLGFFGOLQMYZQTDF33UHZLAVSQ, seg_pos={0 0}, upload=501136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CaBSQgwTru9NXaznMJ2XExKe3yhLoUCvDKFvqvLJGJev4mEPc): protocol: expected piece hash; context canceled; EOF Dec 29 11:05:36.101318363 (571h36m35.972401408s) [noise=true , address=45.91.116.5:4318 , piece_id=PWFEBU4DLSDME6FHXGHE55U6Y7KLGFFGOLQMYZQTDF33UHZLAVSQ, seg_pos={0 0}, upload=501136]: Long tail cancellation due to optimal threshold Dec 29 11:05:36.101621961 (571h36m35.972704996s) [noise=true , address=64.49.13.169:4707 , piece_id=NL2EYMTZOVRQQ4APNVBGE66HJA5QILLU6TFB5TDBGBYU3GO4IRHA, seg_pos={0 0}, upload=501136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:45296->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:45296->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45296->64.49.13.169:4707: use of closed network connection Dec 29 11:05:36.101629084 (571h36m35.972712119s) [noise=true , address=64.49.13.169:4707 , piece_id=NL2EYMTZOVRQQ4APNVBGE66HJA5QILLU6TFB5TDBGBYU3GO4IRHA, seg_pos={0 0}, upload=501136]: Long tail cancellation due to optimal threshold Dec 29 11:05:36.101715780 (571h36m35.972798815s) [noise=true , address=45.91.116.12:4239 , piece_id=5JWXVPMXPTWP53E6PIJULVD55R5S46DQ6KR2ZGGQATDQGZFG4LVA, seg_pos={0 0}, upload=501136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DdwuGVzn339aKBRre1NvC9rUwSbL8v4GLwWj5TZBRP6RvLDnF): protocol: expected piece hash; context canceled; EOF Dec 29 11:05:36.101721320 (571h36m35.972804355s) [noise=true , address=45.91.116.12:4239 , piece_id=5JWXVPMXPTWP53E6PIJULVD55R5S46DQ6KR2ZGGQATDQGZFG4LVA, seg_pos={0 0}, upload=501136]: Long tail cancellation due to optimal threshold Dec 29 11:05:36.101848852 (571h36m35.972931887s) [noise=true , address=145.79.227.40:4069 , piece_id=IGIQ6YEI2FZJMPGG7APVRI4FE5SWBFLSEMUKVLBJ4VJNCEY7I7VQ, seg_pos={0 0}, upload=501136]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 29 11:05:36.101854173 (571h36m35.972937208s) [noise=true , address=145.79.227.40:4069 , piece_id=IGIQ6YEI2FZJMPGG7APVRI4FE5SWBFLSEMUKVLBJ4VJNCEY7I7VQ, seg_pos={0 0}, upload=501136]: Long tail cancellation due to optimal threshold Dec 29 11:05:36.101868661 (571h36m35.972951706s) [seg_pos={0 0}, upload=501136]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 11:05:36.101932713 (571h36m35.973015748s) [upload=501136]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 11:05:36.101937271 (571h36m35.973020316s) [upload=501136]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 11:05:36.403553304 (571h36m36.274636349s) [upload=501136]: Done uploading segments... Dec 29 11:05:36.403573943 (571h36m36.274656978s) [upload=501136]: Upload finished. err: <nil> Duration: 22.846508613s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)