Orbiter History

Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 334715 reports (Page 113 of 6695)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [X....XX......X.X.X.XX.X....X.X..X..XXXXXXX.X.XX...XX.X.XX..X] duration: 30.003157567s trace: https://jaeger.dev.storj.io/trace/693f71387e5574af FAIL (2025-12-27 12:26:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: 4J7UVWNGI73GFWDEL4DZLOCKNFWYFDWI63PKWDFF242OTW3CCARQ) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: VVBYG7HPHNHTBGGD43LQBXZIVHLQ3QG2DWOGXT3SHDGOET6CD72A) context deadline exceeded ======LOGS====== 7 12:26:26.811220668 (524h57m26.682303713s) [upload=460487]: Starting upload Dec 27 12:26:26.811271866 (524h57m26.682354901s) [upload=460487]: Done starting upload Dec 27 12:26:26.811335687 (524h57m26.682418722s) [upload=460487]: Uploading segments... Dec 27 12:26:26.811461797 (524h57m26.682544832s) [upload=460487]: Got next segment. Inline: false Dec 27 12:26:26.811525278 (524h57m26.682608323s) [upload=460487]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 12:26:26.811530708 (524h57m26.682613743s) [upload=460487]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 12:26:27.055686307 (524h57m26.926769352s) [seg_pos={0 0}, upload=460487]: Begin upload segment... Dec 27 12:26:27.055739177 (524h57m26.926822222s) [seg_pos={0 0}, upload=460487]: Done begin upload segment. Dec 27 12:26:27.055773142 (524h57m26.926856187s) [noise=true , address=145.79.227.41:4085 , piece_id=C75IDP2K5QCTHL2STQZRHYPY6BMXWKF47VKUU5QC75EWMJFYICMQ, seg_pos={0 0}, upload=460487]: Uploading piece... Dec 27 12:26:27.055813929 (524h57m26.926896974s) [noise=true , address=145.79.227.43:4157 , piece_id=XYTT6W3IEOQKBNAAA5ZZ7UCD27KSIDVAZ4ICRC5LOBQ5GPLRASQA, seg_pos={0 0}, upload=460487]: Uploading piece... Dec 27 12:26:27.056473976 (524h57m26.927557011s) [noise=true , address=145.79.227.12:4765 , piece_id=QKLV63MFD57VV3OXZKK5J3KLTIBDZNOY3ZRFUX6LJK6B5TSYLEHQ, seg_pos={0 0}, upload=460487]: Uploading piece... Dec 27 12:26:27.056824544 (524h57m26.927907589s) [noise=true , address=145.79.227.32:4834 , piece_id=SJHBG3RXF32SQPNND3KWZH2N6UGOUSTW6O4WQKMSI54V62SU35KQ, seg_pos={0 0}, upload=460487]: Uploading piece... Dec 27 12:26:27.057088166 (524h57m26.928171211s) [noise=true , address=77.90.31.14:4664 , piece_id=7CNQAFT22ZK5ZPPKG3FJ7NJ2325KIRBIBQVWJIQ3AVDD73DIYXLA, seg_pos={0 0}, upload=460487]: Uploading piece... Dec 27 12:26:27.057444384 (524h57m26.928527419s) [noise=true , address=145.79.227.40:4050 , piece_id=5KGW7CJCMVQFBNX2NGP5ZCNT7MHA55PPI2FWN74WVSJ6QALPRQXA, seg_pos={0 0}, upload=460487]: Uploading piece... Dec 27 12:26:27.057709409 (524h57m26.928792444s) [noise=true , address=45.91.116.11:4118 , piece_id=4J7UVWNGI73GFWDEL4DZLOCKNFWYFDWI63PKWDFF242OTW3CCARQ, seg_pos={0 0}, upload=460487]: Uploading piece... Dec 27 12:26:27.058140239 (524h57m26.929223294s) [noise=true , address=45.91.116.14:4189 , piece_id=LDRSQ6FMA4NYVF2O3GNZW5OOPKFDVTT2SZN27HVJ4C27SX4IWX3A, seg_pos={0 0}, upload=460487]: Uploading piece... Dec 27 12:26:27.058487050 (524h57m26.929570095s) [noise=true , address=145.79.227.42:4122 , piece_id=WGT2VOET3SLSP54OSGSKBXH36RSTHXIHRU7FMMITDBPFZXF22PLQ, seg_pos={0 0}, upload=460487]: Uploading piece... Dec 27 12:26:27.058783054 (524h57m26.929866099s) [noise=true , address=45.91.116.12:4227 , piece_id=VVBYG7HPHNHTBGGD43LQBXZIVHLQ3QG2DWOGXT3SHDGOET6CD72A, seg_pos={0 0}, upload=460487]: Uploading piece... Dec 27 12:26:27.059707824 (524h57m26.930790869s) [noise=true , address=145.79.227.20:4438 , piece_id=IFSSO5DROONBQFIE6QVPP3VBAIRAHUYP6OFWQME346B5FBQKZZCQ, seg_pos={0 0}, upload=460487]: Uploading piece... Dec 27 12:26:27.060277680 (524h57m26.931360715s) [noise=true , address=145.79.227.46:4601 , piece_id=JGPSH4U5FW6HAMOKDA54UUS3HJHYUPWQ4SIGL3GTQJXCLGWIQK4Q, seg_pos={0 0}, upload=460487]: Uploading piece... Dec 27 12:26:27.060826504 (524h57m26.931909549s) [noise=true , address=145.79.227.51:4410 , piece_id=ZNOB3BXSHZV37CPIWYZT26YOOD5PC6R7S737BL7PWSOQ7RE4DODQ, seg_pos={0 0}, upload=460487]: Uploading piece... Dec 27 12:26:27.061682244 (524h57m26.932765279s) [noise=true , address=145.79.227.17:4970 , piece_id=A2TEKYIHUOOJUARFUJGNOPFNM36B4VUUW5KRC2LXBXENBVJYRGZA, seg_pos={0 0}, upload=460487]: Uploading piece... Dec 27 12:26:27.062616854 (524h57m26.933699889s) [noise=true , address=145.79.227.11:4729 , piece_id=D3B6F4EX6NY6EFNNRPHQQNV54TDIMMPJJ7KTCIWOJ6FPWO2MIHWA, seg_pos={0 0}, upload=460487]: Uploading piece... Dec 27 12:26:27.063584006 (524h57m26.934667041s) [noise=true , address=77.90.31.18:4805 , piece_id=MJVWBZYVYZEF6EJU67IJO6HUJKQ6DKRALAULBEYX5NLFMW5KWHFQ, seg_pos={0 0}, upload=460487]: Uploading piece... Dec 27 12:26:28.780705995 (524h57m28.65178903s) [upload=460487]: Next returned nil segment Dec 27 12:26:28.780718969 (524h57m28.651802004s) [upload=460487]: Waiting for error group managing segments... Dec 27 12:26:29.056943481 (524h57m28.928026546s) [noise=true , address=145.79.227.11:4729 , piece_id=D3B6F4EX6NY6EFNNRPHQQNV54TDIMMPJJ7KTCIWOJ6FPWO2MIHWA, seg_pos={0 0}, upload=460487]: Done uploading piece. err: <nil> Dec 27 12:26:29.062333275 (524h57m28.93341631s) [noise=true , address=145.79.227.51:4410 , piece_id=ZNOB3BXSHZV37CPIWYZT26YOOD5PC6R7S737BL7PWSOQ7RE4DODQ, seg_pos={0 0}, upload=460487]: Done uploading piece. err: <nil> Dec 27 12:26:29.070700146 (524h57m28.941783191s) [noise=true , address=145.79.227.32:4834 , piece_id=SJHBG3RXF32SQPNND3KWZH2N6UGOUSTW6O4WQKMSI54V62SU35KQ, seg_pos={0 0}, upload=460487]: Done uploading piece. err: <nil> Dec 27 12:26:29.071001480 (524h57m28.942084515s) [noise=true , address=145.79.227.46:4601 , piece_id=JGPSH4U5FW6HAMOKDA54UUS3HJHYUPWQ4SIGL3GTQJXCLGWIQK4Q, seg_pos={0 0}, upload=460487]: Done uploading piece. err: <nil> Dec 27 12:26:29.071344584 (524h57m28.942427619s) [noise=true , address=145.79.227.41:4085 , piece_id=C75IDP2K5QCTHL2STQZRHYPY6BMXWKF47VKUU5QC75EWMJFYICMQ, seg_pos={0 0}, upload=460487]: Done uploading piece. err: <nil> Dec 27 12:26:29.072177109 (524h57m28.943260154s) [noise=true , address=145.79.227.20:4438 , piece_id=IFSSO5DROONBQFIE6QVPP3VBAIRAHUYP6OFWQME346B5FBQKZZCQ, seg_pos={0 0}, upload=460487]: Done uploading piece. err: <nil> Dec 27 12:26:29.086381377 (524h57m28.957464422s) [noise=true , address=145.79.227.12:4765 , piece_id=QKLV63MFD57VV3OXZKK5J3KLTIBDZNOY3ZRFUX6LJK6B5TSYLEHQ, seg_pos={0 0}, upload=460487]: Done uploading piece. err: <nil> Dec 27 12:26:29.087395938 (524h57m28.958478983s) [noise=true , address=145.79.227.17:4970 , piece_id=A2TEKYIHUOOJUARFUJGNOPFNM36B4VUUW5KRC2LXBXENBVJYRGZA, seg_pos={0 0}, upload=460487]: Done uploading piece. err: <nil> Dec 27 12:26:29.092756496 (524h57m28.963839531s) [noise=true , address=145.79.227.42:4122 , piece_id=WGT2VOET3SLSP54OSGSKBXH36RSTHXIHRU7FMMITDBPFZXF22PLQ, seg_pos={0 0}, upload=460487]: Done uploading piece. err: <nil> Dec 27 12:26:29.110890406 (524h57m28.981973451s) [noise=true , address=145.79.227.43:4157 , piece_id=XYTT6W3IEOQKBNAAA5ZZ7UCD27KSIDVAZ4ICRC5LOBQ5GPLRASQA, seg_pos={0 0}, upload=460487]: Done uploading piece. err: <nil> Dec 27 12:26:29.117006562 (524h57m28.988089607s) [noise=true , address=145.79.227.40:4050 , piece_id=5KGW7CJCMVQFBNX2NGP5ZCNT7MHA55PPI2FWN74WVSJ6QALPRQXA, seg_pos={0 0}, upload=460487]: Done uploading piece. err: <nil> Dec 27 12:26:29.352311957 (524h57m29.223394992s) [noise=true , address=45.91.116.11:4118 , piece_id=4J7UVWNGI73GFWDEL4DZLOCKNFWYFDWI63PKWDFF242OTW3CCARQ, seg_pos={0 0}, upload=460487]: Done uploading piece. err: <nil> Dec 27 12:26:29.421530502 (524h57m29.292613547s) [noise=true , address=45.91.116.12:4227 , piece_id=VVBYG7HPHNHTBGGD43LQBXZIVHLQ3QG2DWOGXT3SHDGOET6CD72A, seg_pos={0 0}, upload=460487]: Done uploading piece. err: <nil> Dec 27 12:26:29.421550761 (524h57m29.292633796s) [seg_pos={0 0}, upload=460487]: Segment reached optimal threshold of 13 pieces. Dec 27 12:26:29.421731935 (524h57m29.29281497s) [noise=true , address=77.90.31.14:4664 , piece_id=7CNQAFT22ZK5ZPPKG3FJ7NJ2325KIRBIBQVWJIQ3AVDD73DIYXLA, seg_pos={0 0}, upload=460487]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: write tcp 188.166.225.170:60346->77.90.31.14:4664: use of closed network connection; write tcp 188.166.225.170:60346->77.90.31.14:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60346->77.90.31.14:4664: use of closed network connection Dec 27 12:26:29.421738117 (524h57m29.292821162s) [noise=true , address=77.90.31.14:4664 , piece_id=7CNQAFT22ZK5ZPPKG3FJ7NJ2325KIRBIBQVWJIQ3AVDD73DIYXLA, seg_pos={0 0}, upload=460487]: Long tail cancellation due to optimal threshold Dec 27 12:26:29.421841063 (524h57m29.292924108s) [noise=true , address=77.90.31.18:4805 , piece_id=MJVWBZYVYZEF6EJU67IJO6HUJKQ6DKRALAULBEYX5NLFMW5KWHFQ, seg_pos={0 0}, upload=460487]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Dec 27 12:26:29.421851382 (524h57m29.292934427s) [noise=true , address=77.90.31.18:4805 , piece_id=MJVWBZYVYZEF6EJU67IJO6HUJKQ6DKRALAULBEYX5NLFMW5KWHFQ, seg_pos={0 0}, upload=460487]: Long tail cancellation due to optimal threshold Dec 27 12:26:29.421921596 (524h57m29.293004631s) [noise=true , address=45.91.116.14:4189 , piece_id=LDRSQ6FMA4NYVF2O3GNZW5OOPKFDVTT2SZN27HVJ4C27SX4IWX3A, seg_pos={0 0}, upload=460487]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 27 12:26:29.421928649 (524h57m29.293011684s) [noise=true , address=45.91.116.14:4189 , piece_id=LDRSQ6FMA4NYVF2O3GNZW5OOPKFDVTT2SZN27HVJ4C27SX4IWX3A, seg_pos={0 0}, upload=460487]: Long tail cancellation due to optimal threshold Dec 27 12:26:29.421996949 (524h57m29.293079994s) [seg_pos={0 0}, upload=460487]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 12:26:29.422061232 (524h57m29.293144267s) [upload=460487]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 12:26:29.422065941 (524h57m29.293148976s) [upload=460487]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 12:26:29.714543031 (524h57m29.585626076s) [upload=460487]: Done uploading segments... Dec 27 12:26:29.714586293 (524h57m29.585669328s) [upload=460487]: Upload finished. err: <nil> Duration: 30.003181244s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................................X] duration: 5.001545221s trace: https://jaeger.dev.storj.io/trace/67348817ec207505 FAIL (2025-12-27 12:26:25 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001568515s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [.X....XX......X.X.X.XX.X....X.X..X..XXXXXXX.X.XX...XX.X.XX..] duration: 5.917901589s trace: https://jaeger.dev.storj.io/trace/700bb4fcd2eab89b FAIL (2025-12-27 12:26:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: GJ6ATBRLCKIT4EZUG7KQO4O4JLGO5KOQQRQQWDRS7APLCRMKKN3A) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: ZP4VHZINYM6QYK54TECZOSR7L3OY2ITCZK6Q4KNYG2Z6O4PU7VBQ) context deadline exceeded ======LOGS====== 2:25:23.561407536 (524h56m23.432490581s) [upload=460474]: Starting upload Dec 27 12:25:23.561490054 (524h56m23.432573099s) [upload=460474]: Done starting upload Dec 27 12:25:23.561565618 (524h56m23.432648653s) [upload=460474]: Uploading segments... Dec 27 12:25:23.561956031 (524h56m23.433039066s) [upload=460474]: Got next segment. Inline: false Dec 27 12:25:23.561992430 (524h56m23.433075465s) [upload=460474]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 12:25:23.561997129 (524h56m23.433080164s) [upload=460474]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 12:25:23.809002572 (524h56m23.680085607s) [seg_pos={0 0}, upload=460474]: Begin upload segment... Dec 27 12:25:23.809058028 (524h56m23.680141063s) [seg_pos={0 0}, upload=460474]: Done begin upload segment. Dec 27 12:25:23.809141657 (524h56m23.680224702s) [noise=true , address=145.79.227.8:4331 , piece_id=V2C7EXH3TZDAM523UCML4KG3OOBQSSFQWLMKS55XJUPNQO7K4L6A, seg_pos={0 0}, upload=460474]: Uploading piece... Dec 27 12:25:23.809159902 (524h56m23.680242947s) [noise=true , address=45.91.116.4:4940 , piece_id=GFLSZ64JN7PZW5CCVUJVUCRVDWNPWHB2FGGG2DCTJ3E7RMMOMUPQ, seg_pos={0 0}, upload=460474]: Uploading piece... Dec 27 12:25:23.809850938 (524h56m23.680933973s) [noise=true , address=77.90.31.10:4517 , piece_id=45ELB2DAOFT6A4PAED7GQ2G3NT236V2D4Y7ELBPDYTCPUYW2CBCA, seg_pos={0 0}, upload=460474]: Uploading piece... Dec 27 12:25:23.810431614 (524h56m23.681514659s) [noise=true , address=145.79.227.11:4732 , piece_id=TYUAQYZCS2LGKHP76MT6WR7DQDAHC3OTMFLDCOZZHCVIQYXHHDDQ, seg_pos={0 0}, upload=460474]: Uploading piece... Dec 27 12:25:23.810591077 (524h56m23.681674123s) [noise=true , address=77.90.31.19:4844 , piece_id=JLQVLACUKCHM7TQTUN5JTX6XRJX4XB2E3ZKZ5BXZI52YU7T5AACQ, seg_pos={0 0}, upload=460474]: Uploading piece... Dec 27 12:25:23.810919604 (524h56m23.682002649s) [noise=true , address=145.79.227.4:4186 , piece_id=N6L54KYKNB4T2X35Y5UVXFQEB47VQGEZKAZ2PBQPUPTWAS7FQX7A, seg_pos={0 0}, upload=460474]: Uploading piece... Dec 27 12:25:23.811355213 (524h56m23.682438248s) [noise=true , address=145.79.227.14:4837 , piece_id=CKQQSPHEZ5SIFTKIY76CZUZ4YXFY6W357NNO2ZBYNEC2E5TMXURQ, seg_pos={0 0}, upload=460474]: Uploading piece... Dec 27 12:25:23.811552629 (524h56m23.682635664s) [noise=true , address=145.79.227.2:4114 , piece_id=TZNU3Z42ML6AQY2KIIXUBW5FK24FW74X6HTLTU7GRU2Z2XNDAS5A, seg_pos={0 0}, upload=460474]: Uploading piece... Dec 27 12:25:23.811888267 (524h56m23.682971302s) [noise=true , address=145.79.227.37:4567 , piece_id=VXYROVAGW7LGVZHLWHMS4QPK26C33R5RYCLFGBFSRYNU2N2KGGUA, seg_pos={0 0}, upload=460474]: Uploading piece... Dec 27 12:25:23.812085513 (524h56m23.683168548s) [noise=true , address=77.90.31.17:4770 , piece_id=C5MH2A6JGSJU65YTH3STZYHNLKUHAIZIEFZGS3AZZBYZKJW26XYA, seg_pos={0 0}, upload=460474]: Uploading piece... Dec 27 12:25:23.812298248 (524h56m23.683381283s) [noise=true , address=45.91.116.8:4046 , piece_id=EDKFDCE2BBI7ESNEYQBTXIWEI5OQTJFUDDU7FMHESR63OJF25DBA, seg_pos={0 0}, upload=460474]: Uploading piece... Dec 27 12:25:23.812575256 (524h56m23.683658291s) [noise=true , address=45.91.116.6:4973 , piece_id=GJ6ATBRLCKIT4EZUG7KQO4O4JLGO5KOQQRQQWDRS7APLCRMKKN3A, seg_pos={0 0}, upload=460474]: Uploading piece... Dec 27 12:25:23.812645089 (524h56m23.683728124s) [noise=true , address=45.91.116.13:4153 , piece_id=ZP4VHZINYM6QYK54TECZOSR7L3OY2ITCZK6Q4KNYG2Z6O4PU7VBQ, seg_pos={0 0}, upload=460474]: Uploading piece... Dec 27 12:25:23.812997510 (524h56m23.684080545s) [noise=true , address=145.79.227.45:4230 , piece_id=72XZFEHIPGSWOA6ZEAL6BT4QMXDTBJ4WB6YBWNPMCV6MNACYQIEQ, seg_pos={0 0}, upload=460474]: Uploading piece... Dec 27 12:25:23.813040802 (524h56m23.684123847s) [noise=true , address=145.79.227.32:4834 , piece_id=VMSU4W4VI2IY3OJY7ZCEV2ACR7CMY3NJCOT737W23RQ27PIAPDRQ, seg_pos={0 0}, upload=460474]: Uploading piece... Dec 27 12:25:23.813429102 (524h56m23.684512147s) [noise=true , address=45.91.116.14:4191 , piece_id=KVCIR2EW6QRYVKSL24WJBMYMAD53KYLEDKVV2ZQWRYNSN36M6IDQ, seg_pos={0 0}, upload=460474]: Uploading piece... Dec 27 12:25:25.436331643 (524h56m25.307414678s) [upload=460474]: Next returned nil segment Dec 27 12:25:25.436346481 (524h56m25.307429526s) [upload=460474]: Waiting for error group managing segments... Dec 27 12:25:25.821001775 (524h56m25.69208481s) [noise=true , address=145.79.227.14:4837 , piece_id=CKQQSPHEZ5SIFTKIY76CZUZ4YXFY6W357NNO2ZBYNEC2E5TMXURQ, seg_pos={0 0}, upload=460474]: Done uploading piece. err: <nil> Dec 27 12:25:25.823110079 (524h56m25.694193114s) [noise=true , address=145.79.227.11:4732 , piece_id=TYUAQYZCS2LGKHP76MT6WR7DQDAHC3OTMFLDCOZZHCVIQYXHHDDQ, seg_pos={0 0}, upload=460474]: Done uploading piece. err: <nil> Dec 27 12:25:25.823182386 (524h56m25.694265442s) [noise=true , address=145.79.227.4:4186 , piece_id=N6L54KYKNB4T2X35Y5UVXFQEB47VQGEZKAZ2PBQPUPTWAS7FQX7A, seg_pos={0 0}, upload=460474]: Done uploading piece. err: <nil> Dec 27 12:25:25.823935631 (524h56m25.695018676s) [noise=true , address=145.79.227.37:4567 , piece_id=VXYROVAGW7LGVZHLWHMS4QPK26C33R5RYCLFGBFSRYNU2N2KGGUA, seg_pos={0 0}, upload=460474]: Done uploading piece. err: <nil> Dec 27 12:25:25.825979734 (524h56m25.697062769s) [noise=true , address=145.79.227.32:4834 , piece_id=VMSU4W4VI2IY3OJY7ZCEV2ACR7CMY3NJCOT737W23RQ27PIAPDRQ, seg_pos={0 0}, upload=460474]: Done uploading piece. err: <nil> Dec 27 12:25:25.830251237 (524h56m25.701334272s) [noise=true , address=145.79.227.8:4331 , piece_id=V2C7EXH3TZDAM523UCML4KG3OOBQSSFQWLMKS55XJUPNQO7K4L6A, seg_pos={0 0}, upload=460474]: Done uploading piece. err: <nil> Dec 27 12:25:25.839401300 (524h56m25.710484335s) [noise=true , address=145.79.227.45:4230 , piece_id=72XZFEHIPGSWOA6ZEAL6BT4QMXDTBJ4WB6YBWNPMCV6MNACYQIEQ, seg_pos={0 0}, upload=460474]: Done uploading piece. err: <nil> Dec 27 12:25:25.842589952 (524h56m25.713672997s) [noise=true , address=145.79.227.2:4114 , piece_id=TZNU3Z42ML6AQY2KIIXUBW5FK24FW74X6HTLTU7GRU2Z2XNDAS5A, seg_pos={0 0}, upload=460474]: Done uploading piece. err: <nil> Dec 27 12:25:26.052982638 (524h56m25.924065673s) [noise=true , address=45.91.116.6:4973 , piece_id=GJ6ATBRLCKIT4EZUG7KQO4O4JLGO5KOQQRQQWDRS7APLCRMKKN3A, seg_pos={0 0}, upload=460474]: Done uploading piece. err: <nil> Dec 27 12:25:26.406275751 (524h56m26.277358797s) [noise=true , address=45.91.116.8:4046 , piece_id=EDKFDCE2BBI7ESNEYQBTXIWEI5OQTJFUDDU7FMHESR63OJF25DBA, seg_pos={0 0}, upload=460474]: Done uploading piece. err: <nil> Dec 27 12:25:26.576582346 (524h56m26.447665391s) [noise=true , address=45.91.116.4:4940 , piece_id=GFLSZ64JN7PZW5CCVUJVUCRVDWNPWHB2FGGG2DCTJ3E7RMMOMUPQ, seg_pos={0 0}, upload=460474]: Done uploading piece. err: <nil> Dec 27 12:25:26.764102598 (524h56m26.635185643s) [noise=true , address=77.90.31.10:4517 , piece_id=45ELB2DAOFT6A4PAED7GQ2G3NT236V2D4Y7ELBPDYTCPUYW2CBCA, seg_pos={0 0}, upload=460474]: Done uploading piece. err: <nil> Dec 27 12:25:26.795814976 (524h56m26.666898021s) [noise=true , address=45.91.116.13:4153 , piece_id=ZP4VHZINYM6QYK54TECZOSR7L3OY2ITCZK6Q4KNYG2Z6O4PU7VBQ, seg_pos={0 0}, upload=460474]: Done uploading piece. err: <nil> Dec 27 12:25:26.795837478 (524h56m26.666920523s) [seg_pos={0 0}, upload=460474]: Segment reached optimal threshold of 13 pieces. Dec 27 12:25:26.796043801 (524h56m26.667126836s) [noise=true , address=77.90.31.17:4770 , piece_id=C5MH2A6JGSJU65YTH3STZYHNLKUHAIZIEFZGS3AZZBYZKJW26XYA, seg_pos={0 0}, upload=460474]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 27 12:25:26.796050794 (524h56m26.66713383s) [noise=true , address=77.90.31.17:4770 , piece_id=C5MH2A6JGSJU65YTH3STZYHNLKUHAIZIEFZGS3AZZBYZKJW26XYA, seg_pos={0 0}, upload=460474]: Long tail cancellation due to optimal threshold Dec 27 12:25:26.796184660 (524h56m26.667267695s) [noise=true , address=77.90.31.19:4844 , piece_id=JLQVLACUKCHM7TQTUN5JTX6XRJX4XB2E3ZKZ5BXZI52YU7T5AACQ, seg_pos={0 0}, upload=460474]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 27 12:25:26.796201873 (524h56m26.667284918s) [noise=true , address=77.90.31.19:4844 , piece_id=JLQVLACUKCHM7TQTUN5JTX6XRJX4XB2E3ZKZ5BXZI52YU7T5AACQ, seg_pos={0 0}, upload=460474]: Long tail cancellation due to optimal threshold Dec 27 12:25:26.796376926 (524h56m26.667459971s) [noise=true , address=45.91.116.14:4191 , piece_id=KVCIR2EW6QRYVKSL24WJBMYMAD53KYLEDKVV2ZQWRYNSN36M6IDQ, seg_pos={0 0}, upload=460474]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:56996->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:56996->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56996->45.91.116.14:4191: use of closed network connection Dec 27 12:25:26.796386013 (524h56m26.667469048s) [noise=true , address=45.91.116.14:4191 , piece_id=KVCIR2EW6QRYVKSL24WJBMYMAD53KYLEDKVV2ZQWRYNSN36M6IDQ, seg_pos={0 0}, upload=460474]: Long tail cancellation due to optimal threshold Dec 27 12:25:26.796411522 (524h56m26.667494567s) [seg_pos={0 0}, upload=460474]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 12:25:26.796495381 (524h56m26.667578426s) [upload=460474]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 12:25:26.796502274 (524h56m26.667585319s) [upload=460474]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 12:25:27.095046597 (524h56m26.966129702s) [upload=460474]: Done uploading segments... Dec 27 12:25:27.095069380 (524h56m26.966152425s) [upload=460474]: Upload finished. err: <nil> Duration: 5.917930516s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 7.015109324s trace: https://jaeger.dev.storj.io/trace/578b4df2b1bacee FAIL (2025-12-27 12:26:08 UTC) orbiter warning: context deadline exceeded Duration: 7.015227139s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................X.................X..............] duration: 44.727462021s trace: https://jaeger.dev.storj.io/trace/3b6d882384466243 FAIL (2025-12-27 12:26:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 13QCH91kRMQjVj2ASsVDcyXQ1aapQRVeUm6MMCGimsR6Ns973h, Piece ID: EP2667PIQIHUWPTWD4REAWTQFSC3VR7V7VZQ344XJH4ASSJP4EMA) context deadline exceeded; piecestore: (Node ID: 126Sou5pnMm2eE2FJpv6WYX2VSqJvZZf9rFFJD6hFYDetRD28zu, Piece ID: Y2CDTLJ5S6DCNVTSXNOYT6FU7TSJYREG2BTIDPMOZAQ6FDXHQYAA) context deadline exceeded; piecestore: (Node ID: 12eLKUg2vTnZMrLaraSPEoGXe3jDVBJYyURdcn4f2gCN5pvvCQ8, Piece ID: 3DFZTINQGUUTX3YDEXERPDRZNN5CKAWOMVAQHL23ES2U5YHH2RAA) context deadline exceeded; piecestore: (Node ID: 1X5A7bLqMEQuaMbf9xZ7PDUD4161sz2QqdM9r59y3bsyPBJhaK, Piece ID: 65WTA5P6FPBVAHUJEAX3TIFC23M3A52IZ6FLARKO5RPSSIBZEXWQ) context deadline exceeded ======LOGS====== 38:4484 , piece_id=4US2ZAFTL4NOMWPTAV6ABOM4GE673PJHVJ7C5B3YGYDVFQUPQLXA, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.326171965 (524h44m33.197255s) [noise=true , address=77.90.31.12:4603 , piece_id=Y2CDTLJ5S6DCNVTSXNOYT6FU7TSJYREG2BTIDPMOZAQ6FDXHQYAA, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.326204638 (524h44m33.197287683s) [noise=true , address=77.90.31.13:4644 , piece_id=EP2667PIQIHUWPTWD4REAWTQFSC3VR7V7VZQ344XJH4ASSJP4EMA, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.326588799 (524h44m33.197671844s) [noise=true , address=145.79.227.44:4213 , piece_id=LE27IRRSIIE7T7JQLEMMW4DNJ3XMSRFOPE2DF2XV5XCNWZU3HDEQ, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.326927453 (524h44m33.198010498s) [noise=true , address=77.90.31.15:4721 , piece_id=3A72PLJ5CH6JGOMDL4PUYK3DVIUXRDHVQQA3WA2RW77FXKY46YOQ, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.326992878 (524h44m33.198075923s) [noise=true , address=145.79.227.32:4860 , piece_id=DXII7EZHDNAX6XTRBF6T2N2LTNP5H3DPHQEGDWKN7ZL7V3MVLDXQ, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.327699914 (524h44m33.198782949s) [noise=true , address=45.91.116.12:4258 , piece_id=3DFZTINQGUUTX3YDEXERPDRZNN5CKAWOMVAQHL23ES2U5YHH2RAA, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.327720574 (524h44m33.198803609s) [noise=true , address=145.79.227.34:4914 , piece_id=J7JR3XUPHJI3K4TZ23NECWAGL6YLFURO6MB7QR64L7LCEIBMWUXA, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.328318352 (524h44m33.199401387s) [noise=true , address=77.90.31.16:4746 , piece_id=UJJK5KRSEB3JCBXVUSE4TDHJ5JMIEDIGT4EWETKJCBEE46HSM2VA, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.328412561 (524h44m33.199495616s) [noise=true , address=145.79.227.30:4775 , piece_id=D7SJXV4WU5ZGJVYZHZ2UOVAF72ANYWSOCS5BH5U3Z3EI7MXW536A, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.328792936 (524h44m33.199875981s) [noise=true , address=77.90.31.11:4558 , piece_id=HGCZHZGYN6OPOFJJEIMQG2LXMGA6JDHIDDN3RTRF7XSZ56OL3M7Q, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.329056156 (524h44m33.200139191s) [noise=true , address=77.90.31.21:4905 , piece_id=65WTA5P6FPBVAHUJEAX3TIFC23M3A52IZ6FLARKO5RPSSIBZEXWQ, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.329239847 (524h44m33.200322892s) [noise=true , address=64.49.13.169:4709 , piece_id=XOM2255ZNKHB3I26XEYPYVMPCX4QTYGKL6NLXQOPOZ4PQCELVEAQ, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.329737634 (524h44m33.20082068s) [noise=true , address=145.79.227.40:4072 , piece_id=56RK7LN3GVLMGIYZD6FQ2FU23ADTGNWUYMJUMIHH5HHKATLYOQWQ, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.330431346 (524h44m33.201514391s) [noise=true , address=45.91.116.9:4287 , piece_id=F3TDAQGQWDLWZL2WSKWKMMB77TCNBHLIZVY6NPM5MX2ARUSVW37Q, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:34.611654557 (524h44m34.482737602s) [upload=460306]: Next returned nil segment Dec 27 12:13:34.611669636 (524h44m34.482752681s) [upload=460306]: Waiting for error group managing segments... Dec 27 12:13:34.940178727 (524h44m34.811261762s) [noise=true , address=64.49.13.138:4484 , piece_id=4US2ZAFTL4NOMWPTAV6ABOM4GE673PJHVJ7C5B3YGYDVFQUPQLXA, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.218550166 (524h44m35.089633211s) [noise=true , address=145.79.227.32:4860 , piece_id=DXII7EZHDNAX6XTRBF6T2N2LTNP5H3DPHQEGDWKN7ZL7V3MVLDXQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.226012146 (524h44m35.097095191s) [noise=true , address=145.79.227.34:4914 , piece_id=J7JR3XUPHJI3K4TZ23NECWAGL6YLFURO6MB7QR64L7LCEIBMWUXA, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.400566879 (524h44m35.271649914s) [noise=true , address=145.79.227.33:4891 , piece_id=2R47TJ3SLIMPLH4JZTSINHBZXQ3JJKXVETIMILFATRXEI6DZLJLQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.404347268 (524h44m35.275430303s) [noise=true , address=77.90.31.21:4905 , piece_id=65WTA5P6FPBVAHUJEAX3TIFC23M3A52IZ6FLARKO5RPSSIBZEXWQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.416074639 (524h44m35.287157674s) [noise=true , address=77.90.31.13:4644 , piece_id=EP2667PIQIHUWPTWD4REAWTQFSC3VR7V7VZQ344XJH4ASSJP4EMA, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.452194618 (524h44m35.323277663s) [noise=true , address=45.91.116.9:4287 , piece_id=F3TDAQGQWDLWZL2WSKWKMMB77TCNBHLIZVY6NPM5MX2ARUSVW37Q, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.525559618 (524h44m35.396642663s) [noise=true , address=45.91.116.12:4258 , piece_id=3DFZTINQGUUTX3YDEXERPDRZNN5CKAWOMVAQHL23ES2U5YHH2RAA, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.710902755 (524h44m35.58198579s) [noise=true , address=77.90.31.12:4603 , piece_id=Y2CDTLJ5S6DCNVTSXNOYT6FU7TSJYREG2BTIDPMOZAQ6FDXHQYAA, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.833345128 (524h44m35.704428173s) [noise=true , address=77.90.31.15:4721 , piece_id=3A72PLJ5CH6JGOMDL4PUYK3DVIUXRDHVQQA3WA2RW77FXKY46YOQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:36.102928202 (524h44m35.974011237s) [noise=true , address=145.79.227.44:4213 , piece_id=LE27IRRSIIE7T7JQLEMMW4DNJ3XMSRFOPE2DF2XV5XCNWZU3HDEQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:36.153147128 (524h44m36.024230183s) [noise=true , address=145.79.227.40:4072 , piece_id=56RK7LN3GVLMGIYZD6FQ2FU23ADTGNWUYMJUMIHH5HHKATLYOQWQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:36.208908849 (524h44m36.079991884s) [noise=true , address=145.79.227.30:4775 , piece_id=D7SJXV4WU5ZGJVYZHZ2UOVAF72ANYWSOCS5BH5U3Z3EI7MXW536A, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:36.410529180 (524h44m36.281612215s) [noise=true , address=45.91.116.12:4248 , piece_id=G5SOGWARFYFBXQRSRB5OXZZEKN6BMGN3PQC2RBB7JHN55QUPRXIQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:36.410547656 (524h44m36.281630691s) [seg_pos={0 0}, upload=460306]: Segment reached optimal threshold of 14 pieces. Dec 27 12:13:36.410724162 (524h44m36.281807197s) [noise=true , address=77.90.31.11:4558 , piece_id=HGCZHZGYN6OPOFJJEIMQG2LXMGA6JDHIDDN3RTRF7XSZ56OL3M7Q, seg_pos={0 0}, upload=460306]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mGEP1n1icC2gEkyRVy6BeM4vsrgqYWaQeB5rUtxjrWQjyFhTS): protocol: expected piece hash; context canceled; EOF Dec 27 12:13:36.410730103 (524h44m36.281813138s) [noise=true , address=77.90.31.11:4558 , piece_id=HGCZHZGYN6OPOFJJEIMQG2LXMGA6JDHIDDN3RTRF7XSZ56OL3M7Q, seg_pos={0 0}, upload=460306]: Long tail cancellation due to optimal threshold Dec 27 12:13:36.410916448 (524h44m36.281999483s) [noise=true , address=77.90.31.16:4746 , piece_id=UJJK5KRSEB3JCBXVUSE4TDHJ5JMIEDIGT4EWETKJCBEE46HSM2VA, seg_pos={0 0}, upload=460306]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o): protocol: write tcp 188.166.225.170:42390->77.90.31.16:4746: use of closed network connection; write tcp 188.166.225.170:42390->77.90.31.16:4746: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42390->77.90.31.16:4746: use of closed network connection Dec 27 12:13:36.410922750 (524h44m36.282005785s) [noise=true , address=77.90.31.16:4746 , piece_id=UJJK5KRSEB3JCBXVUSE4TDHJ5JMIEDIGT4EWETKJCBEE46HSM2VA, seg_pos={0 0}, upload=460306]: Long tail cancellation due to optimal threshold Dec 27 12:13:36.411020866 (524h44m36.282103911s) [noise=true , address=193.149.164.143:4699 , piece_id=4A5T3OSNSHNKNXKRWOWVJC222TNAISNRGN3NCXQGVFKNHVDBKXHQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LncNSBhhreg9JTByJPSB7eGUEu88Eyd3jc9pg24VsEjp4FULq): protocol: expected piece hash; context canceled; EOF Dec 27 12:13:36.411036306 (524h44m36.282119341s) [noise=true , address=193.149.164.143:4699 , piece_id=4A5T3OSNSHNKNXKRWOWVJC222TNAISNRGN3NCXQGVFKNHVDBKXHQ, seg_pos={0 0}, upload=460306]: Long tail cancellation due to optimal threshold Dec 27 12:13:36.411086592 (524h44m36.282169637s) [noise=true , address=64.49.13.169:4709 , piece_id=XOM2255ZNKHB3I26XEYPYVMPCX4QTYGKL6NLXQOPOZ4PQCELVEAQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:58550->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:58550->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58550->64.49.13.169:4709: use of closed network connection Dec 27 12:13:36.411093095 (524h44m36.28217613s) [noise=true , address=64.49.13.169:4709 , piece_id=XOM2255ZNKHB3I26XEYPYVMPCX4QTYGKL6NLXQOPOZ4PQCELVEAQ, seg_pos={0 0}, upload=460306]: Long tail cancellation due to optimal threshold Dec 27 12:13:36.411118663 (524h44m36.282201708s) [seg_pos={0 0}, upload=460306]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 12:13:36.411202012 (524h44m36.282285047s) [upload=460306]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 12:13:36.411207102 (524h44m36.282290137s) [upload=460306]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 12:13:36.701956341 (524h44m36.573039386s) [upload=460306]: Done uploading segments... Dec 27 12:13:36.701976790 (524h44m36.573059825s) [upload=460306]: Upload finished. err: <nil> Duration: 44.727539198s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [X.X....XX......X.X.X.XX.X....X.X..X..XXXXXXX.X.XX...XX.X.XX.] duration: 5.698501261s trace: https://jaeger.dev.storj.io/trace/700bb4fcd2eab89b FAIL (2025-12-27 12:25:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: GJ6ATBRLCKIT4EZUG7KQO4O4JLGO5KOQQRQQWDRS7APLCRMKKN3A) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: ZP4VHZINYM6QYK54TECZOSR7L3OY2ITCZK6Q4KNYG2Z6O4PU7VBQ) context deadline exceeded ======LOGS====== 2:25:23.561407536 (524h56m23.432490581s) [upload=460474]: Starting upload Dec 27 12:25:23.561490054 (524h56m23.432573099s) [upload=460474]: Done starting upload Dec 27 12:25:23.561565618 (524h56m23.432648653s) [upload=460474]: Uploading segments... Dec 27 12:25:23.561956031 (524h56m23.433039066s) [upload=460474]: Got next segment. Inline: false Dec 27 12:25:23.561992430 (524h56m23.433075465s) [upload=460474]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 12:25:23.561997129 (524h56m23.433080164s) [upload=460474]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 12:25:23.809002572 (524h56m23.680085607s) [seg_pos={0 0}, upload=460474]: Begin upload segment... Dec 27 12:25:23.809058028 (524h56m23.680141063s) [seg_pos={0 0}, upload=460474]: Done begin upload segment. Dec 27 12:25:23.809141657 (524h56m23.680224702s) [noise=true , address=145.79.227.8:4331 , piece_id=V2C7EXH3TZDAM523UCML4KG3OOBQSSFQWLMKS55XJUPNQO7K4L6A, seg_pos={0 0}, upload=460474]: Uploading piece... Dec 27 12:25:23.809159902 (524h56m23.680242947s) [noise=true , address=45.91.116.4:4940 , piece_id=GFLSZ64JN7PZW5CCVUJVUCRVDWNPWHB2FGGG2DCTJ3E7RMMOMUPQ, seg_pos={0 0}, upload=460474]: Uploading piece... Dec 27 12:25:23.809850938 (524h56m23.680933973s) [noise=true , address=77.90.31.10:4517 , piece_id=45ELB2DAOFT6A4PAED7GQ2G3NT236V2D4Y7ELBPDYTCPUYW2CBCA, seg_pos={0 0}, upload=460474]: Uploading piece... Dec 27 12:25:23.810431614 (524h56m23.681514659s) [noise=true , address=145.79.227.11:4732 , piece_id=TYUAQYZCS2LGKHP76MT6WR7DQDAHC3OTMFLDCOZZHCVIQYXHHDDQ, seg_pos={0 0}, upload=460474]: Uploading piece... Dec 27 12:25:23.810591077 (524h56m23.681674123s) [noise=true , address=77.90.31.19:4844 , piece_id=JLQVLACUKCHM7TQTUN5JTX6XRJX4XB2E3ZKZ5BXZI52YU7T5AACQ, seg_pos={0 0}, upload=460474]: Uploading piece... Dec 27 12:25:23.810919604 (524h56m23.682002649s) [noise=true , address=145.79.227.4:4186 , piece_id=N6L54KYKNB4T2X35Y5UVXFQEB47VQGEZKAZ2PBQPUPTWAS7FQX7A, seg_pos={0 0}, upload=460474]: Uploading piece... Dec 27 12:25:23.811355213 (524h56m23.682438248s) [noise=true , address=145.79.227.14:4837 , piece_id=CKQQSPHEZ5SIFTKIY76CZUZ4YXFY6W357NNO2ZBYNEC2E5TMXURQ, seg_pos={0 0}, upload=460474]: Uploading piece... Dec 27 12:25:23.811552629 (524h56m23.682635664s) [noise=true , address=145.79.227.2:4114 , piece_id=TZNU3Z42ML6AQY2KIIXUBW5FK24FW74X6HTLTU7GRU2Z2XNDAS5A, seg_pos={0 0}, upload=460474]: Uploading piece... Dec 27 12:25:23.811888267 (524h56m23.682971302s) [noise=true , address=145.79.227.37:4567 , piece_id=VXYROVAGW7LGVZHLWHMS4QPK26C33R5RYCLFGBFSRYNU2N2KGGUA, seg_pos={0 0}, upload=460474]: Uploading piece... Dec 27 12:25:23.812085513 (524h56m23.683168548s) [noise=true , address=77.90.31.17:4770 , piece_id=C5MH2A6JGSJU65YTH3STZYHNLKUHAIZIEFZGS3AZZBYZKJW26XYA, seg_pos={0 0}, upload=460474]: Uploading piece... Dec 27 12:25:23.812298248 (524h56m23.683381283s) [noise=true , address=45.91.116.8:4046 , piece_id=EDKFDCE2BBI7ESNEYQBTXIWEI5OQTJFUDDU7FMHESR63OJF25DBA, seg_pos={0 0}, upload=460474]: Uploading piece... Dec 27 12:25:23.812575256 (524h56m23.683658291s) [noise=true , address=45.91.116.6:4973 , piece_id=GJ6ATBRLCKIT4EZUG7KQO4O4JLGO5KOQQRQQWDRS7APLCRMKKN3A, seg_pos={0 0}, upload=460474]: Uploading piece... Dec 27 12:25:23.812645089 (524h56m23.683728124s) [noise=true , address=45.91.116.13:4153 , piece_id=ZP4VHZINYM6QYK54TECZOSR7L3OY2ITCZK6Q4KNYG2Z6O4PU7VBQ, seg_pos={0 0}, upload=460474]: Uploading piece... Dec 27 12:25:23.812997510 (524h56m23.684080545s) [noise=true , address=145.79.227.45:4230 , piece_id=72XZFEHIPGSWOA6ZEAL6BT4QMXDTBJ4WB6YBWNPMCV6MNACYQIEQ, seg_pos={0 0}, upload=460474]: Uploading piece... Dec 27 12:25:23.813040802 (524h56m23.684123847s) [noise=true , address=145.79.227.32:4834 , piece_id=VMSU4W4VI2IY3OJY7ZCEV2ACR7CMY3NJCOT737W23RQ27PIAPDRQ, seg_pos={0 0}, upload=460474]: Uploading piece... Dec 27 12:25:23.813429102 (524h56m23.684512147s) [noise=true , address=45.91.116.14:4191 , piece_id=KVCIR2EW6QRYVKSL24WJBMYMAD53KYLEDKVV2ZQWRYNSN36M6IDQ, seg_pos={0 0}, upload=460474]: Uploading piece... Dec 27 12:25:25.436331643 (524h56m25.307414678s) [upload=460474]: Next returned nil segment Dec 27 12:25:25.436346481 (524h56m25.307429526s) [upload=460474]: Waiting for error group managing segments... Dec 27 12:25:25.821001775 (524h56m25.69208481s) [noise=true , address=145.79.227.14:4837 , piece_id=CKQQSPHEZ5SIFTKIY76CZUZ4YXFY6W357NNO2ZBYNEC2E5TMXURQ, seg_pos={0 0}, upload=460474]: Done uploading piece. err: <nil> Dec 27 12:25:25.823110079 (524h56m25.694193114s) [noise=true , address=145.79.227.11:4732 , piece_id=TYUAQYZCS2LGKHP76MT6WR7DQDAHC3OTMFLDCOZZHCVIQYXHHDDQ, seg_pos={0 0}, upload=460474]: Done uploading piece. err: <nil> Dec 27 12:25:25.823182386 (524h56m25.694265442s) [noise=true , address=145.79.227.4:4186 , piece_id=N6L54KYKNB4T2X35Y5UVXFQEB47VQGEZKAZ2PBQPUPTWAS7FQX7A, seg_pos={0 0}, upload=460474]: Done uploading piece. err: <nil> Dec 27 12:25:25.823935631 (524h56m25.695018676s) [noise=true , address=145.79.227.37:4567 , piece_id=VXYROVAGW7LGVZHLWHMS4QPK26C33R5RYCLFGBFSRYNU2N2KGGUA, seg_pos={0 0}, upload=460474]: Done uploading piece. err: <nil> Dec 27 12:25:25.825979734 (524h56m25.697062769s) [noise=true , address=145.79.227.32:4834 , piece_id=VMSU4W4VI2IY3OJY7ZCEV2ACR7CMY3NJCOT737W23RQ27PIAPDRQ, seg_pos={0 0}, upload=460474]: Done uploading piece. err: <nil> Dec 27 12:25:25.830251237 (524h56m25.701334272s) [noise=true , address=145.79.227.8:4331 , piece_id=V2C7EXH3TZDAM523UCML4KG3OOBQSSFQWLMKS55XJUPNQO7K4L6A, seg_pos={0 0}, upload=460474]: Done uploading piece. err: <nil> Dec 27 12:25:25.839401300 (524h56m25.710484335s) [noise=true , address=145.79.227.45:4230 , piece_id=72XZFEHIPGSWOA6ZEAL6BT4QMXDTBJ4WB6YBWNPMCV6MNACYQIEQ, seg_pos={0 0}, upload=460474]: Done uploading piece. err: <nil> Dec 27 12:25:25.842589952 (524h56m25.713672997s) [noise=true , address=145.79.227.2:4114 , piece_id=TZNU3Z42ML6AQY2KIIXUBW5FK24FW74X6HTLTU7GRU2Z2XNDAS5A, seg_pos={0 0}, upload=460474]: Done uploading piece. err: <nil> Dec 27 12:25:26.052982638 (524h56m25.924065673s) [noise=true , address=45.91.116.6:4973 , piece_id=GJ6ATBRLCKIT4EZUG7KQO4O4JLGO5KOQQRQQWDRS7APLCRMKKN3A, seg_pos={0 0}, upload=460474]: Done uploading piece. err: <nil> Dec 27 12:25:26.406275751 (524h56m26.277358797s) [noise=true , address=45.91.116.8:4046 , piece_id=EDKFDCE2BBI7ESNEYQBTXIWEI5OQTJFUDDU7FMHESR63OJF25DBA, seg_pos={0 0}, upload=460474]: Done uploading piece. err: <nil> Dec 27 12:25:26.576582346 (524h56m26.447665391s) [noise=true , address=45.91.116.4:4940 , piece_id=GFLSZ64JN7PZW5CCVUJVUCRVDWNPWHB2FGGG2DCTJ3E7RMMOMUPQ, seg_pos={0 0}, upload=460474]: Done uploading piece. err: <nil> Dec 27 12:25:26.764102598 (524h56m26.635185643s) [noise=true , address=77.90.31.10:4517 , piece_id=45ELB2DAOFT6A4PAED7GQ2G3NT236V2D4Y7ELBPDYTCPUYW2CBCA, seg_pos={0 0}, upload=460474]: Done uploading piece. err: <nil> Dec 27 12:25:26.795814976 (524h56m26.666898021s) [noise=true , address=45.91.116.13:4153 , piece_id=ZP4VHZINYM6QYK54TECZOSR7L3OY2ITCZK6Q4KNYG2Z6O4PU7VBQ, seg_pos={0 0}, upload=460474]: Done uploading piece. err: <nil> Dec 27 12:25:26.795837478 (524h56m26.666920523s) [seg_pos={0 0}, upload=460474]: Segment reached optimal threshold of 13 pieces. Dec 27 12:25:26.796043801 (524h56m26.667126836s) [noise=true , address=77.90.31.17:4770 , piece_id=C5MH2A6JGSJU65YTH3STZYHNLKUHAIZIEFZGS3AZZBYZKJW26XYA, seg_pos={0 0}, upload=460474]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 27 12:25:26.796050794 (524h56m26.66713383s) [noise=true , address=77.90.31.17:4770 , piece_id=C5MH2A6JGSJU65YTH3STZYHNLKUHAIZIEFZGS3AZZBYZKJW26XYA, seg_pos={0 0}, upload=460474]: Long tail cancellation due to optimal threshold Dec 27 12:25:26.796184660 (524h56m26.667267695s) [noise=true , address=77.90.31.19:4844 , piece_id=JLQVLACUKCHM7TQTUN5JTX6XRJX4XB2E3ZKZ5BXZI52YU7T5AACQ, seg_pos={0 0}, upload=460474]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 27 12:25:26.796201873 (524h56m26.667284918s) [noise=true , address=77.90.31.19:4844 , piece_id=JLQVLACUKCHM7TQTUN5JTX6XRJX4XB2E3ZKZ5BXZI52YU7T5AACQ, seg_pos={0 0}, upload=460474]: Long tail cancellation due to optimal threshold Dec 27 12:25:26.796376926 (524h56m26.667459971s) [noise=true , address=45.91.116.14:4191 , piece_id=KVCIR2EW6QRYVKSL24WJBMYMAD53KYLEDKVV2ZQWRYNSN36M6IDQ, seg_pos={0 0}, upload=460474]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:56996->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:56996->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56996->45.91.116.14:4191: use of closed network connection Dec 27 12:25:26.796386013 (524h56m26.667469048s) [noise=true , address=45.91.116.14:4191 , piece_id=KVCIR2EW6QRYVKSL24WJBMYMAD53KYLEDKVV2ZQWRYNSN36M6IDQ, seg_pos={0 0}, upload=460474]: Long tail cancellation due to optimal threshold Dec 27 12:25:26.796411522 (524h56m26.667494567s) [seg_pos={0 0}, upload=460474]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 12:25:26.796495381 (524h56m26.667578426s) [upload=460474]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 12:25:26.796502274 (524h56m26.667585319s) [upload=460474]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 12:25:27.095046597 (524h56m26.966129702s) [upload=460474]: Done uploading segments... Dec 27 12:25:27.095069380 (524h56m26.966152425s) [upload=460474]: Upload finished. err: <nil> Duration: 5.698519064s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [XX.X....XX......X.X.X.XX.X....X.X..X..XXXXXXX.X.XX...XX.X.XX] duration: 30.002504567s trace: https://jaeger.dev.storj.io/trace/700bb4fcd2eab89b FAIL (2025-12-27 12:25:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: GJ6ATBRLCKIT4EZUG7KQO4O4JLGO5KOQQRQQWDRS7APLCRMKKN3A) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: ZP4VHZINYM6QYK54TECZOSR7L3OY2ITCZK6Q4KNYG2Z6O4PU7VBQ) context deadline exceeded ======LOGS====== 2:25:23.561407536 (524h56m23.432490581s) [upload=460474]: Starting upload Dec 27 12:25:23.561490054 (524h56m23.432573099s) [upload=460474]: Done starting upload Dec 27 12:25:23.561565618 (524h56m23.432648653s) [upload=460474]: Uploading segments... Dec 27 12:25:23.561956031 (524h56m23.433039066s) [upload=460474]: Got next segment. Inline: false Dec 27 12:25:23.561992430 (524h56m23.433075465s) [upload=460474]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 12:25:23.561997129 (524h56m23.433080164s) [upload=460474]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 12:25:23.809002572 (524h56m23.680085607s) [seg_pos={0 0}, upload=460474]: Begin upload segment... Dec 27 12:25:23.809058028 (524h56m23.680141063s) [seg_pos={0 0}, upload=460474]: Done begin upload segment. Dec 27 12:25:23.809141657 (524h56m23.680224702s) [noise=true , address=145.79.227.8:4331 , piece_id=V2C7EXH3TZDAM523UCML4KG3OOBQSSFQWLMKS55XJUPNQO7K4L6A, seg_pos={0 0}, upload=460474]: Uploading piece... Dec 27 12:25:23.809159902 (524h56m23.680242947s) [noise=true , address=45.91.116.4:4940 , piece_id=GFLSZ64JN7PZW5CCVUJVUCRVDWNPWHB2FGGG2DCTJ3E7RMMOMUPQ, seg_pos={0 0}, upload=460474]: Uploading piece... Dec 27 12:25:23.809850938 (524h56m23.680933973s) [noise=true , address=77.90.31.10:4517 , piece_id=45ELB2DAOFT6A4PAED7GQ2G3NT236V2D4Y7ELBPDYTCPUYW2CBCA, seg_pos={0 0}, upload=460474]: Uploading piece... Dec 27 12:25:23.810431614 (524h56m23.681514659s) [noise=true , address=145.79.227.11:4732 , piece_id=TYUAQYZCS2LGKHP76MT6WR7DQDAHC3OTMFLDCOZZHCVIQYXHHDDQ, seg_pos={0 0}, upload=460474]: Uploading piece... Dec 27 12:25:23.810591077 (524h56m23.681674123s) [noise=true , address=77.90.31.19:4844 , piece_id=JLQVLACUKCHM7TQTUN5JTX6XRJX4XB2E3ZKZ5BXZI52YU7T5AACQ, seg_pos={0 0}, upload=460474]: Uploading piece... Dec 27 12:25:23.810919604 (524h56m23.682002649s) [noise=true , address=145.79.227.4:4186 , piece_id=N6L54KYKNB4T2X35Y5UVXFQEB47VQGEZKAZ2PBQPUPTWAS7FQX7A, seg_pos={0 0}, upload=460474]: Uploading piece... Dec 27 12:25:23.811355213 (524h56m23.682438248s) [noise=true , address=145.79.227.14:4837 , piece_id=CKQQSPHEZ5SIFTKIY76CZUZ4YXFY6W357NNO2ZBYNEC2E5TMXURQ, seg_pos={0 0}, upload=460474]: Uploading piece... Dec 27 12:25:23.811552629 (524h56m23.682635664s) [noise=true , address=145.79.227.2:4114 , piece_id=TZNU3Z42ML6AQY2KIIXUBW5FK24FW74X6HTLTU7GRU2Z2XNDAS5A, seg_pos={0 0}, upload=460474]: Uploading piece... Dec 27 12:25:23.811888267 (524h56m23.682971302s) [noise=true , address=145.79.227.37:4567 , piece_id=VXYROVAGW7LGVZHLWHMS4QPK26C33R5RYCLFGBFSRYNU2N2KGGUA, seg_pos={0 0}, upload=460474]: Uploading piece... Dec 27 12:25:23.812085513 (524h56m23.683168548s) [noise=true , address=77.90.31.17:4770 , piece_id=C5MH2A6JGSJU65YTH3STZYHNLKUHAIZIEFZGS3AZZBYZKJW26XYA, seg_pos={0 0}, upload=460474]: Uploading piece... Dec 27 12:25:23.812298248 (524h56m23.683381283s) [noise=true , address=45.91.116.8:4046 , piece_id=EDKFDCE2BBI7ESNEYQBTXIWEI5OQTJFUDDU7FMHESR63OJF25DBA, seg_pos={0 0}, upload=460474]: Uploading piece... Dec 27 12:25:23.812575256 (524h56m23.683658291s) [noise=true , address=45.91.116.6:4973 , piece_id=GJ6ATBRLCKIT4EZUG7KQO4O4JLGO5KOQQRQQWDRS7APLCRMKKN3A, seg_pos={0 0}, upload=460474]: Uploading piece... Dec 27 12:25:23.812645089 (524h56m23.683728124s) [noise=true , address=45.91.116.13:4153 , piece_id=ZP4VHZINYM6QYK54TECZOSR7L3OY2ITCZK6Q4KNYG2Z6O4PU7VBQ, seg_pos={0 0}, upload=460474]: Uploading piece... Dec 27 12:25:23.812997510 (524h56m23.684080545s) [noise=true , address=145.79.227.45:4230 , piece_id=72XZFEHIPGSWOA6ZEAL6BT4QMXDTBJ4WB6YBWNPMCV6MNACYQIEQ, seg_pos={0 0}, upload=460474]: Uploading piece... Dec 27 12:25:23.813040802 (524h56m23.684123847s) [noise=true , address=145.79.227.32:4834 , piece_id=VMSU4W4VI2IY3OJY7ZCEV2ACR7CMY3NJCOT737W23RQ27PIAPDRQ, seg_pos={0 0}, upload=460474]: Uploading piece... Dec 27 12:25:23.813429102 (524h56m23.684512147s) [noise=true , address=45.91.116.14:4191 , piece_id=KVCIR2EW6QRYVKSL24WJBMYMAD53KYLEDKVV2ZQWRYNSN36M6IDQ, seg_pos={0 0}, upload=460474]: Uploading piece... Dec 27 12:25:25.436331643 (524h56m25.307414678s) [upload=460474]: Next returned nil segment Dec 27 12:25:25.436346481 (524h56m25.307429526s) [upload=460474]: Waiting for error group managing segments... Dec 27 12:25:25.821001775 (524h56m25.69208481s) [noise=true , address=145.79.227.14:4837 , piece_id=CKQQSPHEZ5SIFTKIY76CZUZ4YXFY6W357NNO2ZBYNEC2E5TMXURQ, seg_pos={0 0}, upload=460474]: Done uploading piece. err: <nil> Dec 27 12:25:25.823110079 (524h56m25.694193114s) [noise=true , address=145.79.227.11:4732 , piece_id=TYUAQYZCS2LGKHP76MT6WR7DQDAHC3OTMFLDCOZZHCVIQYXHHDDQ, seg_pos={0 0}, upload=460474]: Done uploading piece. err: <nil> Dec 27 12:25:25.823182386 (524h56m25.694265442s) [noise=true , address=145.79.227.4:4186 , piece_id=N6L54KYKNB4T2X35Y5UVXFQEB47VQGEZKAZ2PBQPUPTWAS7FQX7A, seg_pos={0 0}, upload=460474]: Done uploading piece. err: <nil> Dec 27 12:25:25.823935631 (524h56m25.695018676s) [noise=true , address=145.79.227.37:4567 , piece_id=VXYROVAGW7LGVZHLWHMS4QPK26C33R5RYCLFGBFSRYNU2N2KGGUA, seg_pos={0 0}, upload=460474]: Done uploading piece. err: <nil> Dec 27 12:25:25.825979734 (524h56m25.697062769s) [noise=true , address=145.79.227.32:4834 , piece_id=VMSU4W4VI2IY3OJY7ZCEV2ACR7CMY3NJCOT737W23RQ27PIAPDRQ, seg_pos={0 0}, upload=460474]: Done uploading piece. err: <nil> Dec 27 12:25:25.830251237 (524h56m25.701334272s) [noise=true , address=145.79.227.8:4331 , piece_id=V2C7EXH3TZDAM523UCML4KG3OOBQSSFQWLMKS55XJUPNQO7K4L6A, seg_pos={0 0}, upload=460474]: Done uploading piece. err: <nil> Dec 27 12:25:25.839401300 (524h56m25.710484335s) [noise=true , address=145.79.227.45:4230 , piece_id=72XZFEHIPGSWOA6ZEAL6BT4QMXDTBJ4WB6YBWNPMCV6MNACYQIEQ, seg_pos={0 0}, upload=460474]: Done uploading piece. err: <nil> Dec 27 12:25:25.842589952 (524h56m25.713672997s) [noise=true , address=145.79.227.2:4114 , piece_id=TZNU3Z42ML6AQY2KIIXUBW5FK24FW74X6HTLTU7GRU2Z2XNDAS5A, seg_pos={0 0}, upload=460474]: Done uploading piece. err: <nil> Dec 27 12:25:26.052982638 (524h56m25.924065673s) [noise=true , address=45.91.116.6:4973 , piece_id=GJ6ATBRLCKIT4EZUG7KQO4O4JLGO5KOQQRQQWDRS7APLCRMKKN3A, seg_pos={0 0}, upload=460474]: Done uploading piece. err: <nil> Dec 27 12:25:26.406275751 (524h56m26.277358797s) [noise=true , address=45.91.116.8:4046 , piece_id=EDKFDCE2BBI7ESNEYQBTXIWEI5OQTJFUDDU7FMHESR63OJF25DBA, seg_pos={0 0}, upload=460474]: Done uploading piece. err: <nil> Dec 27 12:25:26.576582346 (524h56m26.447665391s) [noise=true , address=45.91.116.4:4940 , piece_id=GFLSZ64JN7PZW5CCVUJVUCRVDWNPWHB2FGGG2DCTJ3E7RMMOMUPQ, seg_pos={0 0}, upload=460474]: Done uploading piece. err: <nil> Dec 27 12:25:26.764102598 (524h56m26.635185643s) [noise=true , address=77.90.31.10:4517 , piece_id=45ELB2DAOFT6A4PAED7GQ2G3NT236V2D4Y7ELBPDYTCPUYW2CBCA, seg_pos={0 0}, upload=460474]: Done uploading piece. err: <nil> Dec 27 12:25:26.795814976 (524h56m26.666898021s) [noise=true , address=45.91.116.13:4153 , piece_id=ZP4VHZINYM6QYK54TECZOSR7L3OY2ITCZK6Q4KNYG2Z6O4PU7VBQ, seg_pos={0 0}, upload=460474]: Done uploading piece. err: <nil> Dec 27 12:25:26.795837478 (524h56m26.666920523s) [seg_pos={0 0}, upload=460474]: Segment reached optimal threshold of 13 pieces. Dec 27 12:25:26.796043801 (524h56m26.667126836s) [noise=true , address=77.90.31.17:4770 , piece_id=C5MH2A6JGSJU65YTH3STZYHNLKUHAIZIEFZGS3AZZBYZKJW26XYA, seg_pos={0 0}, upload=460474]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 27 12:25:26.796050794 (524h56m26.66713383s) [noise=true , address=77.90.31.17:4770 , piece_id=C5MH2A6JGSJU65YTH3STZYHNLKUHAIZIEFZGS3AZZBYZKJW26XYA, seg_pos={0 0}, upload=460474]: Long tail cancellation due to optimal threshold Dec 27 12:25:26.796184660 (524h56m26.667267695s) [noise=true , address=77.90.31.19:4844 , piece_id=JLQVLACUKCHM7TQTUN5JTX6XRJX4XB2E3ZKZ5BXZI52YU7T5AACQ, seg_pos={0 0}, upload=460474]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 27 12:25:26.796201873 (524h56m26.667284918s) [noise=true , address=77.90.31.19:4844 , piece_id=JLQVLACUKCHM7TQTUN5JTX6XRJX4XB2E3ZKZ5BXZI52YU7T5AACQ, seg_pos={0 0}, upload=460474]: Long tail cancellation due to optimal threshold Dec 27 12:25:26.796376926 (524h56m26.667459971s) [noise=true , address=45.91.116.14:4191 , piece_id=KVCIR2EW6QRYVKSL24WJBMYMAD53KYLEDKVV2ZQWRYNSN36M6IDQ, seg_pos={0 0}, upload=460474]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:56996->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:56996->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56996->45.91.116.14:4191: use of closed network connection Dec 27 12:25:26.796386013 (524h56m26.667469048s) [noise=true , address=45.91.116.14:4191 , piece_id=KVCIR2EW6QRYVKSL24WJBMYMAD53KYLEDKVV2ZQWRYNSN36M6IDQ, seg_pos={0 0}, upload=460474]: Long tail cancellation due to optimal threshold Dec 27 12:25:26.796411522 (524h56m26.667494567s) [seg_pos={0 0}, upload=460474]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 12:25:26.796495381 (524h56m26.667578426s) [upload=460474]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 12:25:26.796502274 (524h56m26.667585319s) [upload=460474]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 12:25:27.095046597 (524h56m26.966129702s) [upload=460474]: Done uploading segments... Dec 27 12:25:27.095069380 (524h56m26.966152425s) [upload=460474]: Upload finished. err: <nil> Duration: 30.002529695s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [.XX.X....XX......X.X.X.XX.X....X.X..X..XXXXXXX.X.XX...XX.X.X] duration: 30.002993451s trace: https://jaeger.dev.storj.io/trace/f0f77a52dceedd6 FAIL (2025-12-27 12:25:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== (524h55m49.191207687s) [upload=460465]: Done starting upload Dec 27 12:24:49.320234111 (524h55m49.191317146s) [upload=460465]: Uploading segments... Dec 27 12:24:49.320342607 (524h55m49.191425642s) [upload=460465]: Got next segment. Inline: false Dec 27 12:24:49.320387082 (524h55m49.191470117s) [upload=460465]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 12:24:49.320392453 (524h55m49.191475498s) [upload=460465]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 12:24:49.565787408 (524h55m49.436870454s) [seg_pos={0 0}, upload=460465]: Begin upload segment... Dec 27 12:24:49.565961981 (524h55m49.437045016s) [seg_pos={0 0}, upload=460465]: Done begin upload segment. Dec 27 12:24:49.565994864 (524h55m49.437077899s) [noise=true , address=77.90.31.19:4842 , piece_id=TFCRGR2OFZHYDTNJREMVJLQTH5F4UBEG6PFGX6RUZUPIR3QIXQKQ, seg_pos={0 0}, upload=460465]: Uploading piece... Dec 27 12:24:49.566760712 (524h55m49.437843747s) [noise=true , address=45.91.116.14:4189 , piece_id=Q2FJOV4O2HAW3AQIP7X5BKL2SPJSGP32ZQRULYSOP2CHL52R4QVA, seg_pos={0 0}, upload=460465]: Uploading piece... Dec 27 12:24:49.567018232 (524h55m49.438101267s) [noise=true , address=77.90.31.13:4629 , piece_id=Q7WBQVZK5I7KBUMJAHUTQC3GA6U25726MRI3A5N5RK4CCGC2LZXQ, seg_pos={0 0}, upload=460465]: Uploading piece... Dec 27 12:24:49.567116930 (524h55m49.438199975s) [noise=true , address=77.90.31.21:4880 , piece_id=WUBJ6YQ2CDK6UKXPBGOWJLSG5RWD2JGJ5HWTPP2SRPKSFHJCQDUA, seg_pos={0 0}, upload=460465]: Uploading piece... Dec 27 12:24:49.567430347 (524h55m49.438513392s) [noise=true , address=145.79.227.50:4373 , piece_id=RYMYNOWRWUF7DWLGQ2Z6VPRXPFTIRZY7T6W5VN36G2BB5MJGCIFQ, seg_pos={0 0}, upload=460465]: Uploading piece... Dec 27 12:24:49.567606252 (524h55m49.438689297s) [noise=true , address=45.91.116.5:4300 , piece_id=BYLPL4ZQXIHNRHTL3XCFVDMEKJLD52GLLD5JWRJ3GJMPNEHZJJLA, seg_pos={0 0}, upload=460465]: Uploading piece... Dec 27 12:24:49.567691834 (524h55m49.438774869s) [noise=true , address=77.90.31.18:4805 , piece_id=IH47AKGAOLDYF2WDSXXWSCZZFSU3WPDDKOVL7BNISNAG75UO7BCQ, seg_pos={0 0}, upload=460465]: Uploading piece... Dec 27 12:24:49.567950106 (524h55m49.439033142s) [noise=true , address=77.90.31.11:4553 , piece_id=OW5LOBTOVDWYFYNJ3VQI5ZA7PL5UHJZBO5M5KS3GQUJGDHTKYLAA, seg_pos={0 0}, upload=460465]: Uploading piece... Dec 27 12:24:49.568024368 (524h55m49.439107413s) [noise=true , address=77.90.31.10:4518 , piece_id=CY34AHVHCIGKV4G5ODMJEOCZNVAMVW4LWHXZLSUBZZHHJ4OHHW3Q, seg_pos={0 0}, upload=460465]: Uploading piece... Dec 27 12:24:49.568239377 (524h55m49.439322412s) [noise=true , address=77.90.31.14:4662 , piece_id=DLJGQ3GTYI32AAOODJYRCWVA75NBTK7CR7QECCYAK54O6KISPSPQ, seg_pos={0 0}, upload=460465]: Uploading piece... Dec 27 12:24:49.568464276 (524h55m49.439547311s) [noise=true , address=145.79.227.39:4015 , piece_id=A7JBDTRBCQWFSJXEDDEHVUHJPBEOUGDDL6ZJAMNFJBZM4NFNDQZA, seg_pos={0 0}, upload=460465]: Uploading piece... Dec 27 12:24:49.568503671 (524h55m49.439586716s) [noise=true , address=145.79.227.51:4411 , piece_id=UCR2VVUUUTK6Q6BIBLRNYR6Q5H5LRT5POAOGEVT4HXR524ZD6JVA, seg_pos={0 0}, upload=460465]: Uploading piece... Dec 27 12:24:49.568752885 (524h55m49.43983592s) [noise=true , address=45.91.116.7:4011 , piece_id=JYFMIJP4MWNTI2T4CSJVIFXYXVFO2H3WSGLT66N2WASPQCMQMRZQ, seg_pos={0 0}, upload=460465]: Uploading piece... Dec 27 12:24:49.568893944 (524h55m49.439976979s) [noise=true , address=45.91.116.9:4261 , piece_id=ABPLDZDFH26CH2GYRERN25DKGSNC5XYPFAVKOQNMRMV5FXHTGMTA, seg_pos={0 0}, upload=460465]: Uploading piece... Dec 27 12:24:49.569010405 (524h55m49.44009347s) [noise=true , address=45.91.116.10:4083 , piece_id=4PA5P7BYYCE4QEODJH4BHRTTAM6YKZ7YZGM3DLPDJ4KDLZFXTOBQ, seg_pos={0 0}, upload=460465]: Uploading piece... Dec 27 12:24:49.569257866 (524h55m49.440340911s) [noise=true , address=45.91.116.4:4940 , piece_id=TXLZSJPI3KZLF7S44L2SLZ4FENI5SZNLTNYCU7OLL62ZVMJX5LUQ, seg_pos={0 0}, upload=460465]: Uploading piece... Dec 27 12:24:51.289726541 (524h55m51.160809576s) [upload=460465]: Next returned nil segment Dec 27 12:24:51.289748283 (524h55m51.160831318s) [upload=460465]: Waiting for error group managing segments... Dec 27 12:24:51.577173783 (524h55m51.448256828s) [noise=true , address=145.79.227.51:4411 , piece_id=UCR2VVUUUTK6Q6BIBLRNYR6Q5H5LRT5POAOGEVT4HXR524ZD6JVA, seg_pos={0 0}, upload=460465]: Done uploading piece. err: <nil> Dec 27 12:24:51.598792949 (524h55m51.469875984s) [noise=true , address=145.79.227.50:4373 , piece_id=RYMYNOWRWUF7DWLGQ2Z6VPRXPFTIRZY7T6W5VN36G2BB5MJGCIFQ, seg_pos={0 0}, upload=460465]: Done uploading piece. err: <nil> Dec 27 12:24:51.604491020 (524h55m51.475574055s) [noise=true , address=145.79.227.39:4015 , piece_id=A7JBDTRBCQWFSJXEDDEHVUHJPBEOUGDDL6ZJAMNFJBZM4NFNDQZA, seg_pos={0 0}, upload=460465]: Done uploading piece. err: <nil> Dec 27 12:24:51.792205742 (524h55m51.663288788s) [noise=true , address=77.90.31.19:4842 , piece_id=TFCRGR2OFZHYDTNJREMVJLQTH5F4UBEG6PFGX6RUZUPIR3QIXQKQ, seg_pos={0 0}, upload=460465]: Done uploading piece. err: <nil> Dec 27 12:24:51.810977427 (524h55m51.682060472s) [noise=true , address=45.91.116.14:4189 , piece_id=Q2FJOV4O2HAW3AQIP7X5BKL2SPJSGP32ZQRULYSOP2CHL52R4QVA, seg_pos={0 0}, upload=460465]: Done uploading piece. err: <nil> Dec 27 12:24:51.838659138 (524h55m51.709742183s) [noise=true , address=45.91.116.5:4300 , piece_id=BYLPL4ZQXIHNRHTL3XCFVDMEKJLD52GLLD5JWRJ3GJMPNEHZJJLA, seg_pos={0 0}, upload=460465]: Done uploading piece. err: <nil> Dec 27 12:24:51.867795218 (524h55m51.738878264s) [noise=true , address=77.90.31.21:4880 , piece_id=WUBJ6YQ2CDK6UKXPBGOWJLSG5RWD2JGJ5HWTPP2SRPKSFHJCQDUA, seg_pos={0 0}, upload=460465]: Done uploading piece. err: <nil> Dec 27 12:24:51.917773807 (524h55m51.788856862s) [noise=true , address=45.91.116.10:4083 , piece_id=4PA5P7BYYCE4QEODJH4BHRTTAM6YKZ7YZGM3DLPDJ4KDLZFXTOBQ, seg_pos={0 0}, upload=460465]: Done uploading piece. err: <nil> Dec 27 12:24:51.930321761 (524h55m51.801404796s) [noise=true , address=45.91.116.9:4261 , piece_id=ABPLDZDFH26CH2GYRERN25DKGSNC5XYPFAVKOQNMRMV5FXHTGMTA, seg_pos={0 0}, upload=460465]: Done uploading piece. err: <nil> Dec 27 12:24:51.950430000 (524h55m51.821513035s) [noise=true , address=77.90.31.14:4662 , piece_id=DLJGQ3GTYI32AAOODJYRCWVA75NBTK7CR7QECCYAK54O6KISPSPQ, seg_pos={0 0}, upload=460465]: Done uploading piece. err: <nil> Dec 27 12:24:51.959978703 (524h55m51.831061738s) [noise=true , address=77.90.31.13:4629 , piece_id=Q7WBQVZK5I7KBUMJAHUTQC3GA6U25726MRI3A5N5RK4CCGC2LZXQ, seg_pos={0 0}, upload=460465]: Done uploading piece. err: <nil> Dec 27 12:24:51.967056040 (524h55m51.838139075s) [noise=true , address=45.91.116.4:4940 , piece_id=TXLZSJPI3KZLF7S44L2SLZ4FENI5SZNLTNYCU7OLL62ZVMJX5LUQ, seg_pos={0 0}, upload=460465]: Done uploading piece. err: <nil> Dec 27 12:24:51.985368410 (524h55m51.856451455s) [noise=true , address=45.91.116.7:4011 , piece_id=JYFMIJP4MWNTI2T4CSJVIFXYXVFO2H3WSGLT66N2WASPQCMQMRZQ, seg_pos={0 0}, upload=460465]: Done uploading piece. err: <nil> Dec 27 12:24:51.985388076 (524h55m51.856471112s) [seg_pos={0 0}, upload=460465]: Segment reached optimal threshold of 13 pieces. Dec 27 12:24:51.985568780 (524h55m51.856651825s) [noise=true , address=77.90.31.11:4553 , piece_id=OW5LOBTOVDWYFYNJ3VQI5ZA7PL5UHJZBO5M5KS3GQUJGDHTKYLAA, seg_pos={0 0}, upload=460465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:34264->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:34264->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34264->77.90.31.11:4553: use of closed network connection Dec 27 12:24:51.985575294 (524h55m51.856658339s) [noise=true , address=77.90.31.11:4553 , piece_id=OW5LOBTOVDWYFYNJ3VQI5ZA7PL5UHJZBO5M5KS3GQUJGDHTKYLAA, seg_pos={0 0}, upload=460465]: Long tail cancellation due to optimal threshold Dec 27 12:24:51.985879222 (524h55m51.856962257s) [noise=true , address=77.90.31.10:4518 , piece_id=CY34AHVHCIGKV4G5ODMJEOCZNVAMVW4LWHXZLSUBZZHHJ4OHHW3Q, seg_pos={0 0}, upload=460465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 27 12:24:51.985886095 (524h55m51.85696914s) [noise=true , address=77.90.31.10:4518 , piece_id=CY34AHVHCIGKV4G5ODMJEOCZNVAMVW4LWHXZLSUBZZHHJ4OHHW3Q, seg_pos={0 0}, upload=460465]: Long tail cancellation due to optimal threshold Dec 27 12:24:51.986030190 (524h55m51.857113235s) [noise=true , address=77.90.31.18:4805 , piece_id=IH47AKGAOLDYF2WDSXXWSCZZFSU3WPDDKOVL7BNISNAG75UO7BCQ, seg_pos={0 0}, upload=460465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: read tcp 188.166.225.170:56812->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 12:24:51.986053313 (524h55m51.857136358s) [noise=true , address=77.90.31.18:4805 , piece_id=IH47AKGAOLDYF2WDSXXWSCZZFSU3WPDDKOVL7BNISNAG75UO7BCQ, seg_pos={0 0}, upload=460465]: Long tail cancellation due to optimal threshold Dec 27 12:24:51.986157872 (524h55m51.857240917s) [seg_pos={0 0}, upload=460465]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 12:24:51.986298901 (524h55m51.857381946s) [upload=460465]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 12:24:51.986304743 (524h55m51.857387778s) [upload=460465]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 12:24:52.276080257 (524h55m52.147163302s) [upload=460465]: Done uploading segments... Dec 27 12:24:52.276105444 (524h55m52.147188489s) [upload=460465]: Upload finished. err: <nil> Duration: 30.003015674s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................X.................X.............] duration: 51.666181654s trace: https://jaeger.dev.storj.io/trace/3b6d882384466243 FAIL (2025-12-27 12:25:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 13QCH91kRMQjVj2ASsVDcyXQ1aapQRVeUm6MMCGimsR6Ns973h, Piece ID: EP2667PIQIHUWPTWD4REAWTQFSC3VR7V7VZQ344XJH4ASSJP4EMA) context deadline exceeded; piecestore: (Node ID: 126Sou5pnMm2eE2FJpv6WYX2VSqJvZZf9rFFJD6hFYDetRD28zu, Piece ID: Y2CDTLJ5S6DCNVTSXNOYT6FU7TSJYREG2BTIDPMOZAQ6FDXHQYAA) context deadline exceeded; piecestore: (Node ID: 12eLKUg2vTnZMrLaraSPEoGXe3jDVBJYyURdcn4f2gCN5pvvCQ8, Piece ID: 3DFZTINQGUUTX3YDEXERPDRZNN5CKAWOMVAQHL23ES2U5YHH2RAA) context deadline exceeded; piecestore: (Node ID: 1X5A7bLqMEQuaMbf9xZ7PDUD4161sz2QqdM9r59y3bsyPBJhaK, Piece ID: 65WTA5P6FPBVAHUJEAX3TIFC23M3A52IZ6FLARKO5RPSSIBZEXWQ) context deadline exceeded ======LOGS====== 38:4484 , piece_id=4US2ZAFTL4NOMWPTAV6ABOM4GE673PJHVJ7C5B3YGYDVFQUPQLXA, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.326171965 (524h44m33.197255s) [noise=true , address=77.90.31.12:4603 , piece_id=Y2CDTLJ5S6DCNVTSXNOYT6FU7TSJYREG2BTIDPMOZAQ6FDXHQYAA, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.326204638 (524h44m33.197287683s) [noise=true , address=77.90.31.13:4644 , piece_id=EP2667PIQIHUWPTWD4REAWTQFSC3VR7V7VZQ344XJH4ASSJP4EMA, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.326588799 (524h44m33.197671844s) [noise=true , address=145.79.227.44:4213 , piece_id=LE27IRRSIIE7T7JQLEMMW4DNJ3XMSRFOPE2DF2XV5XCNWZU3HDEQ, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.326927453 (524h44m33.198010498s) [noise=true , address=77.90.31.15:4721 , piece_id=3A72PLJ5CH6JGOMDL4PUYK3DVIUXRDHVQQA3WA2RW77FXKY46YOQ, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.326992878 (524h44m33.198075923s) [noise=true , address=145.79.227.32:4860 , piece_id=DXII7EZHDNAX6XTRBF6T2N2LTNP5H3DPHQEGDWKN7ZL7V3MVLDXQ, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.327699914 (524h44m33.198782949s) [noise=true , address=45.91.116.12:4258 , piece_id=3DFZTINQGUUTX3YDEXERPDRZNN5CKAWOMVAQHL23ES2U5YHH2RAA, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.327720574 (524h44m33.198803609s) [noise=true , address=145.79.227.34:4914 , piece_id=J7JR3XUPHJI3K4TZ23NECWAGL6YLFURO6MB7QR64L7LCEIBMWUXA, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.328318352 (524h44m33.199401387s) [noise=true , address=77.90.31.16:4746 , piece_id=UJJK5KRSEB3JCBXVUSE4TDHJ5JMIEDIGT4EWETKJCBEE46HSM2VA, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.328412561 (524h44m33.199495616s) [noise=true , address=145.79.227.30:4775 , piece_id=D7SJXV4WU5ZGJVYZHZ2UOVAF72ANYWSOCS5BH5U3Z3EI7MXW536A, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.328792936 (524h44m33.199875981s) [noise=true , address=77.90.31.11:4558 , piece_id=HGCZHZGYN6OPOFJJEIMQG2LXMGA6JDHIDDN3RTRF7XSZ56OL3M7Q, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.329056156 (524h44m33.200139191s) [noise=true , address=77.90.31.21:4905 , piece_id=65WTA5P6FPBVAHUJEAX3TIFC23M3A52IZ6FLARKO5RPSSIBZEXWQ, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.329239847 (524h44m33.200322892s) [noise=true , address=64.49.13.169:4709 , piece_id=XOM2255ZNKHB3I26XEYPYVMPCX4QTYGKL6NLXQOPOZ4PQCELVEAQ, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.329737634 (524h44m33.20082068s) [noise=true , address=145.79.227.40:4072 , piece_id=56RK7LN3GVLMGIYZD6FQ2FU23ADTGNWUYMJUMIHH5HHKATLYOQWQ, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.330431346 (524h44m33.201514391s) [noise=true , address=45.91.116.9:4287 , piece_id=F3TDAQGQWDLWZL2WSKWKMMB77TCNBHLIZVY6NPM5MX2ARUSVW37Q, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:34.611654557 (524h44m34.482737602s) [upload=460306]: Next returned nil segment Dec 27 12:13:34.611669636 (524h44m34.482752681s) [upload=460306]: Waiting for error group managing segments... Dec 27 12:13:34.940178727 (524h44m34.811261762s) [noise=true , address=64.49.13.138:4484 , piece_id=4US2ZAFTL4NOMWPTAV6ABOM4GE673PJHVJ7C5B3YGYDVFQUPQLXA, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.218550166 (524h44m35.089633211s) [noise=true , address=145.79.227.32:4860 , piece_id=DXII7EZHDNAX6XTRBF6T2N2LTNP5H3DPHQEGDWKN7ZL7V3MVLDXQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.226012146 (524h44m35.097095191s) [noise=true , address=145.79.227.34:4914 , piece_id=J7JR3XUPHJI3K4TZ23NECWAGL6YLFURO6MB7QR64L7LCEIBMWUXA, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.400566879 (524h44m35.271649914s) [noise=true , address=145.79.227.33:4891 , piece_id=2R47TJ3SLIMPLH4JZTSINHBZXQ3JJKXVETIMILFATRXEI6DZLJLQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.404347268 (524h44m35.275430303s) [noise=true , address=77.90.31.21:4905 , piece_id=65WTA5P6FPBVAHUJEAX3TIFC23M3A52IZ6FLARKO5RPSSIBZEXWQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.416074639 (524h44m35.287157674s) [noise=true , address=77.90.31.13:4644 , piece_id=EP2667PIQIHUWPTWD4REAWTQFSC3VR7V7VZQ344XJH4ASSJP4EMA, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.452194618 (524h44m35.323277663s) [noise=true , address=45.91.116.9:4287 , piece_id=F3TDAQGQWDLWZL2WSKWKMMB77TCNBHLIZVY6NPM5MX2ARUSVW37Q, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.525559618 (524h44m35.396642663s) [noise=true , address=45.91.116.12:4258 , piece_id=3DFZTINQGUUTX3YDEXERPDRZNN5CKAWOMVAQHL23ES2U5YHH2RAA, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.710902755 (524h44m35.58198579s) [noise=true , address=77.90.31.12:4603 , piece_id=Y2CDTLJ5S6DCNVTSXNOYT6FU7TSJYREG2BTIDPMOZAQ6FDXHQYAA, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.833345128 (524h44m35.704428173s) [noise=true , address=77.90.31.15:4721 , piece_id=3A72PLJ5CH6JGOMDL4PUYK3DVIUXRDHVQQA3WA2RW77FXKY46YOQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:36.102928202 (524h44m35.974011237s) [noise=true , address=145.79.227.44:4213 , piece_id=LE27IRRSIIE7T7JQLEMMW4DNJ3XMSRFOPE2DF2XV5XCNWZU3HDEQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:36.153147128 (524h44m36.024230183s) [noise=true , address=145.79.227.40:4072 , piece_id=56RK7LN3GVLMGIYZD6FQ2FU23ADTGNWUYMJUMIHH5HHKATLYOQWQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:36.208908849 (524h44m36.079991884s) [noise=true , address=145.79.227.30:4775 , piece_id=D7SJXV4WU5ZGJVYZHZ2UOVAF72ANYWSOCS5BH5U3Z3EI7MXW536A, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:36.410529180 (524h44m36.281612215s) [noise=true , address=45.91.116.12:4248 , piece_id=G5SOGWARFYFBXQRSRB5OXZZEKN6BMGN3PQC2RBB7JHN55QUPRXIQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:36.410547656 (524h44m36.281630691s) [seg_pos={0 0}, upload=460306]: Segment reached optimal threshold of 14 pieces. Dec 27 12:13:36.410724162 (524h44m36.281807197s) [noise=true , address=77.90.31.11:4558 , piece_id=HGCZHZGYN6OPOFJJEIMQG2LXMGA6JDHIDDN3RTRF7XSZ56OL3M7Q, seg_pos={0 0}, upload=460306]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mGEP1n1icC2gEkyRVy6BeM4vsrgqYWaQeB5rUtxjrWQjyFhTS): protocol: expected piece hash; context canceled; EOF Dec 27 12:13:36.410730103 (524h44m36.281813138s) [noise=true , address=77.90.31.11:4558 , piece_id=HGCZHZGYN6OPOFJJEIMQG2LXMGA6JDHIDDN3RTRF7XSZ56OL3M7Q, seg_pos={0 0}, upload=460306]: Long tail cancellation due to optimal threshold Dec 27 12:13:36.410916448 (524h44m36.281999483s) [noise=true , address=77.90.31.16:4746 , piece_id=UJJK5KRSEB3JCBXVUSE4TDHJ5JMIEDIGT4EWETKJCBEE46HSM2VA, seg_pos={0 0}, upload=460306]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o): protocol: write tcp 188.166.225.170:42390->77.90.31.16:4746: use of closed network connection; write tcp 188.166.225.170:42390->77.90.31.16:4746: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42390->77.90.31.16:4746: use of closed network connection Dec 27 12:13:36.410922750 (524h44m36.282005785s) [noise=true , address=77.90.31.16:4746 , piece_id=UJJK5KRSEB3JCBXVUSE4TDHJ5JMIEDIGT4EWETKJCBEE46HSM2VA, seg_pos={0 0}, upload=460306]: Long tail cancellation due to optimal threshold Dec 27 12:13:36.411020866 (524h44m36.282103911s) [noise=true , address=193.149.164.143:4699 , piece_id=4A5T3OSNSHNKNXKRWOWVJC222TNAISNRGN3NCXQGVFKNHVDBKXHQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LncNSBhhreg9JTByJPSB7eGUEu88Eyd3jc9pg24VsEjp4FULq): protocol: expected piece hash; context canceled; EOF Dec 27 12:13:36.411036306 (524h44m36.282119341s) [noise=true , address=193.149.164.143:4699 , piece_id=4A5T3OSNSHNKNXKRWOWVJC222TNAISNRGN3NCXQGVFKNHVDBKXHQ, seg_pos={0 0}, upload=460306]: Long tail cancellation due to optimal threshold Dec 27 12:13:36.411086592 (524h44m36.282169637s) [noise=true , address=64.49.13.169:4709 , piece_id=XOM2255ZNKHB3I26XEYPYVMPCX4QTYGKL6NLXQOPOZ4PQCELVEAQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:58550->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:58550->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58550->64.49.13.169:4709: use of closed network connection Dec 27 12:13:36.411093095 (524h44m36.28217613s) [noise=true , address=64.49.13.169:4709 , piece_id=XOM2255ZNKHB3I26XEYPYVMPCX4QTYGKL6NLXQOPOZ4PQCELVEAQ, seg_pos={0 0}, upload=460306]: Long tail cancellation due to optimal threshold Dec 27 12:13:36.411118663 (524h44m36.282201708s) [seg_pos={0 0}, upload=460306]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 12:13:36.411202012 (524h44m36.282285047s) [upload=460306]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 12:13:36.411207102 (524h44m36.282290137s) [upload=460306]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 12:13:36.701956341 (524h44m36.573039386s) [upload=460306]: Done uploading segments... Dec 27 12:13:36.701976790 (524h44m36.573059825s) [upload=460306]: Upload finished. err: <nil> Duration: 51.666198726s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 7.909332905s trace: https://jaeger.dev.storj.io/trace/578b4df2b1bacee FAIL (2025-12-27 12:25:16 UTC) orbiter warning: context deadline exceeded Duration: 7.909359376s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 3.457060481s trace: https://jaeger.dev.storj.io/trace/12569321705fed16 FAIL (2025-12-27 12:25:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.457092651s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 2.331087053s trace: https://jaeger.dev.storj.io/trace/6dd289079d7aa430 FAIL (2025-12-27 12:24:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.331112992s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 6.501726528s trace: https://jaeger.dev.storj.io/trace/578b4df2b1bacee FAIL (2025-12-27 12:24:38 UTC) orbiter warning: context deadline exceeded Duration: 6.501778277s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 2.35453911s trace: https://jaeger.dev.storj.io/trace/6dd289079d7aa430 FAIL (2025-12-27 12:24:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.354573616s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [..XX.X....XX......X.X.X.XX.X....X.X..X..XXXXXXX.X.XX...XX.X.] duration: 5.612813215s trace: https://jaeger.dev.storj.io/trace/6cf6706bb6e7ded8 FAIL (2025-12-27 12:24:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: NW63JDCXYKJAS7F6Z7M5P75VJXLXJF6YAKEWD4OCBQEQSO6LXJTQ) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: G5F6ZX6LUCOA2B3XTWY7GDKHO5KF5GRI6U26J3VF2S76A25SQ62Q) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: F43KDFWHQLIW62AKO35HBR7GLDH37I3PVM6KCWPRS6AYFGTWSOEA) context deadline exceeded ======LOGS====== Dec 27 12:23:51.817164147 (524h54m51.688247182s) [upload=460449]: Starting upload Dec 27 12:23:51.817252766 (524h54m51.688335841s) [upload=460449]: Done starting upload Dec 27 12:23:51.817304173 (524h54m51.688387208s) [upload=460449]: Uploading segments... Dec 27 12:23:51.817519284 (524h54m51.688602329s) [upload=460449]: Got next segment. Inline: false Dec 27 12:23:51.817551424 (524h54m51.688634459s) [upload=460449]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 12:23:51.817556605 (524h54m51.68863964s) [upload=460449]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 12:23:52.067545508 (524h54m51.938628543s) [seg_pos={0 0}, upload=460449]: Begin upload segment... Dec 27 12:23:52.067599912 (524h54m51.938682947s) [seg_pos={0 0}, upload=460449]: Done begin upload segment. Dec 27 12:23:52.067630540 (524h54m51.938713575s) [noise=true , address=145.79.227.46:4603 , piece_id=C2P6AAAZLZTHHV27KWBX7Q2CMAMGCSIY7DSHELEM66YAEQTUZXGA, seg_pos={0 0}, upload=460449]: Uploading piece... Dec 27 12:23:52.067659305 (524h54m51.93874234s) [noise=true , address=45.91.116.11:4118 , piece_id=NW63JDCXYKJAS7F6Z7M5P75VJXLXJF6YAKEWD4OCBQEQSO6LXJTQ, seg_pos={0 0}, upload=460449]: Uploading piece... Dec 27 12:23:52.068259889 (524h54m51.939342924s) [noise=true , address=145.79.227.11:4730 , piece_id=OYQQAZVGMZ4UMBCEMFVA4TCC3MYPZ7V66QJNPQTDLZKL2NPTGJXA, seg_pos={0 0}, upload=460449]: Uploading piece... Dec 27 12:23:52.068582152 (524h54m51.939665187s) [noise=true , address=145.79.227.47:4265 , piece_id=JGO5ULVAKZQNV6SISXFVVYZEEI3ISZ6ZJITH3CE3G5QBYJ3QHKRQ, seg_pos={0 0}, upload=460449]: Uploading piece... Dec 27 12:23:52.068880159 (524h54m51.939963194s) [noise=true , address=45.91.116.12:4227 , piece_id=SJZW2EZSFQFMREN2CPBTZNB5EU5LFU2SYBE2HPSLPE5WORVNKN5A, seg_pos={0 0}, upload=460449]: Uploading piece... Dec 27 12:23:52.070426054 (524h54m51.941509099s) [noise=true , address=45.91.116.10:4082 , piece_id=EHOHDYBYMBL5LL3YGW65XWMHZREQWFWQ7BKIWIPJAK275MOL7XWQ, seg_pos={0 0}, upload=460449]: Uploading piece... Dec 27 12:23:52.070802531 (524h54m51.941885566s) [noise=true , address=145.79.227.14:4837 , piece_id=RIIIVSMSAWEUWIGRHZYG2IIP5OJ6P4GRAZBZDT36A7Q7BKGE7DGA, seg_pos={0 0}, upload=460449]: Uploading piece... Dec 27 12:23:52.071178927 (524h54m51.942261962s) [noise=true , address=145.79.227.25:4617 , piece_id=DXHLYI5LJNAK2UASJAMY4CHJI6X43NJCXMGP53ZROCHQV33RIU5Q, seg_pos={0 0}, upload=460449]: Uploading piece... Dec 27 12:23:52.072786448 (524h54m51.943869483s) [noise=true , address=45.91.116.14:4189 , piece_id=I44OLDAZY6NLSYVZB233KVXOITGS2RXCFNVAUENGLUOQXBERAGUA, seg_pos={0 0}, upload=460449]: Uploading piece... Dec 27 12:23:52.074481455 (524h54m51.9455645s) [noise=true , address=145.79.227.2:4113 , piece_id=BY47A2PLISIYPP3SBPNLSD2SYABHSTEZQZHHXOXXFXDHFKPY66LA, seg_pos={0 0}, upload=460449]: Uploading piece... Dec 27 12:23:52.078972246 (524h54m51.950055282s) [noise=true , address=145.79.227.39:4015 , piece_id=665EBMIGZKO755SVRAA6GG4GO3JWX62QAJBVKIUQILFY37XN5OEQ, seg_pos={0 0}, upload=460449]: Uploading piece... Dec 27 12:23:52.081374140 (524h54m51.952457175s) [noise=true , address=145.79.227.42:4122 , piece_id=SSIQ3D3UAH5R6VYBUE5JKCARE3RMIUGJ2HC4AJF2BDKKDDCGVPIA, seg_pos={0 0}, upload=460449]: Uploading piece... Dec 27 12:23:52.082854279 (524h54m51.953937314s) [noise=true , address=45.91.116.9:4264 , piece_id=G5F6ZX6LUCOA2B3XTWY7GDKHO5KF5GRI6U26J3VF2S76A25SQ62Q, seg_pos={0 0}, upload=460449]: Uploading piece... Dec 27 12:23:52.083655705 (524h54m51.95473874s) [noise=true , address=145.79.227.7:4295 , piece_id=ZMJD5AAMLLJAL622YTPCEGDRFUHOT5GM4WCWJRLIK7JYMOYNGCPA, seg_pos={0 0}, upload=460449]: Uploading piece... Dec 27 12:23:52.085709496 (524h54m51.956792531s) [noise=true , address=145.79.227.51:4409 , piece_id=4FLC4A3ON6QQCVNQ3DQUB35LL7IVOY6PT7JHOWFBKTETD6W6IUOA, seg_pos={0 0}, upload=460449]: Uploading piece... Dec 27 12:23:52.089855871 (524h54m51.960938916s) [noise=true , address=45.91.116.8:4048 , piece_id=F43KDFWHQLIW62AKO35HBR7GLDH37I3PVM6KCWPRS6AYFGTWSOEA, seg_pos={0 0}, upload=460449]: Uploading piece... Dec 27 12:23:53.791147820 (524h54m53.662230865s) [upload=460449]: Next returned nil segment Dec 27 12:23:53.791162068 (524h54m53.662245113s) [upload=460449]: Waiting for error group managing segments... Dec 27 12:23:54.083030148 (524h54m53.954113193s) [noise=true , address=145.79.227.42:4122 , piece_id=SSIQ3D3UAH5R6VYBUE5JKCARE3RMIUGJ2HC4AJF2BDKKDDCGVPIA, seg_pos={0 0}, upload=460449]: Done uploading piece. err: <nil> Dec 27 12:23:54.085281024 (524h54m53.956364069s) [noise=true , address=145.79.227.2:4113 , piece_id=BY47A2PLISIYPP3SBPNLSD2SYABHSTEZQZHHXOXXFXDHFKPY66LA, seg_pos={0 0}, upload=460449]: Done uploading piece. err: <nil> Dec 27 12:23:54.086108228 (524h54m53.957191273s) [noise=true , address=145.79.227.47:4265 , piece_id=JGO5ULVAKZQNV6SISXFVVYZEEI3ISZ6ZJITH3CE3G5QBYJ3QHKRQ, seg_pos={0 0}, upload=460449]: Done uploading piece. err: <nil> Dec 27 12:23:54.089593665 (524h54m53.96067671s) [noise=true , address=145.79.227.7:4295 , piece_id=ZMJD5AAMLLJAL622YTPCEGDRFUHOT5GM4WCWJRLIK7JYMOYNGCPA, seg_pos={0 0}, upload=460449]: Done uploading piece. err: <nil> Dec 27 12:23:54.098052552 (524h54m53.969135587s) [noise=true , address=145.79.227.14:4837 , piece_id=RIIIVSMSAWEUWIGRHZYG2IIP5OJ6P4GRAZBZDT36A7Q7BKGE7DGA, seg_pos={0 0}, upload=460449]: Done uploading piece. err: <nil> Dec 27 12:23:54.102026420 (524h54m53.973109455s) [noise=true , address=145.79.227.46:4603 , piece_id=C2P6AAAZLZTHHV27KWBX7Q2CMAMGCSIY7DSHELEM66YAEQTUZXGA, seg_pos={0 0}, upload=460449]: Done uploading piece. err: <nil> Dec 27 12:23:54.102841882 (524h54m53.973924947s) [noise=true , address=145.79.227.25:4617 , piece_id=DXHLYI5LJNAK2UASJAMY4CHJI6X43NJCXMGP53ZROCHQV33RIU5Q, seg_pos={0 0}, upload=460449]: Done uploading piece. err: <nil> Dec 27 12:23:54.109331379 (524h54m53.980414424s) [noise=true , address=145.79.227.51:4409 , piece_id=4FLC4A3ON6QQCVNQ3DQUB35LL7IVOY6PT7JHOWFBKTETD6W6IUOA, seg_pos={0 0}, upload=460449]: Done uploading piece. err: <nil> Dec 27 12:23:54.111015927 (524h54m53.982098972s) [noise=true , address=145.79.227.39:4015 , piece_id=665EBMIGZKO755SVRAA6GG4GO3JWX62QAJBVKIUQILFY37XN5OEQ, seg_pos={0 0}, upload=460449]: Done uploading piece. err: <nil> Dec 27 12:23:54.111572667 (524h54m53.982655703s) [noise=true , address=145.79.227.11:4730 , piece_id=OYQQAZVGMZ4UMBCEMFVA4TCC3MYPZ7V66QJNPQTDLZKL2NPTGJXA, seg_pos={0 0}, upload=460449]: Done uploading piece. err: <nil> Dec 27 12:23:54.326498475 (524h54m54.19758152s) [noise=true , address=45.91.116.11:4118 , piece_id=NW63JDCXYKJAS7F6Z7M5P75VJXLXJF6YAKEWD4OCBQEQSO6LXJTQ, seg_pos={0 0}, upload=460449]: Done uploading piece. err: <nil> Dec 27 12:23:54.432110971 (524h54m54.303194016s) [noise=true , address=45.91.116.9:4264 , piece_id=G5F6ZX6LUCOA2B3XTWY7GDKHO5KF5GRI6U26J3VF2S76A25SQ62Q, seg_pos={0 0}, upload=460449]: Done uploading piece. err: <nil> Dec 27 12:23:54.726187126 (524h54m54.597270171s) [noise=true , address=45.91.116.8:4048 , piece_id=F43KDFWHQLIW62AKO35HBR7GLDH37I3PVM6KCWPRS6AYFGTWSOEA, seg_pos={0 0}, upload=460449]: Done uploading piece. err: <nil> Dec 27 12:23:54.726218175 (524h54m54.59730122s) [seg_pos={0 0}, upload=460449]: Segment reached optimal threshold of 13 pieces. Dec 27 12:23:54.726598468 (524h54m54.597681513s) [noise=true , address=45.91.116.14:4189 , piece_id=I44OLDAZY6NLSYVZB233KVXOITGS2RXCFNVAUENGLUOQXBERAGUA, seg_pos={0 0}, upload=460449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 27 12:23:54.726615822 (524h54m54.597698857s) [noise=true , address=45.91.116.14:4189 , piece_id=I44OLDAZY6NLSYVZB233KVXOITGS2RXCFNVAUENGLUOQXBERAGUA, seg_pos={0 0}, upload=460449]: Long tail cancellation due to optimal threshold Dec 27 12:23:54.726647993 (524h54m54.597731038s) [noise=true , address=45.91.116.10:4082 , piece_id=EHOHDYBYMBL5LL3YGW65XWMHZREQWFWQ7BKIWIPJAK275MOL7XWQ, seg_pos={0 0}, upload=460449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: read tcp 188.166.225.170:54420->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 12:23:54.726658943 (524h54m54.597741979s) [noise=true , address=45.91.116.10:4082 , piece_id=EHOHDYBYMBL5LL3YGW65XWMHZREQWFWQ7BKIWIPJAK275MOL7XWQ, seg_pos={0 0}, upload=460449]: Long tail cancellation due to optimal threshold Dec 27 12:23:54.726840459 (524h54m54.597923504s) [noise=true , address=45.91.116.12:4227 , piece_id=SJZW2EZSFQFMREN2CPBTZNB5EU5LFU2SYBE2HPSLPE5WORVNKN5A, seg_pos={0 0}, upload=460449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: read tcp 188.166.225.170:45966->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 12:23:54.726848325 (524h54m54.59793137s) [noise=true , address=45.91.116.12:4227 , piece_id=SJZW2EZSFQFMREN2CPBTZNB5EU5LFU2SYBE2HPSLPE5WORVNKN5A, seg_pos={0 0}, upload=460449]: Long tail cancellation due to optimal threshold Dec 27 12:23:54.726871178 (524h54m54.597954223s) [seg_pos={0 0}, upload=460449]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 12:23:54.726957032 (524h54m54.598040077s) [upload=460449]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 12:23:54.726963303 (524h54m54.598046348s) [upload=460449]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 12:23:55.033521536 (524h54m54.904604571s) [upload=460449]: Done uploading segments... Dec 27 12:23:55.033543488 (524h54m54.904626533s) [upload=460449]: Upload finished. err: <nil> Duration: 5.612843202s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................X.................X............] duration: 49.79280478s trace: https://jaeger.dev.storj.io/trace/3b6d882384466243 FAIL (2025-12-27 12:24:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 13QCH91kRMQjVj2ASsVDcyXQ1aapQRVeUm6MMCGimsR6Ns973h, Piece ID: EP2667PIQIHUWPTWD4REAWTQFSC3VR7V7VZQ344XJH4ASSJP4EMA) context deadline exceeded; piecestore: (Node ID: 126Sou5pnMm2eE2FJpv6WYX2VSqJvZZf9rFFJD6hFYDetRD28zu, Piece ID: Y2CDTLJ5S6DCNVTSXNOYT6FU7TSJYREG2BTIDPMOZAQ6FDXHQYAA) context deadline exceeded; piecestore: (Node ID: 12eLKUg2vTnZMrLaraSPEoGXe3jDVBJYyURdcn4f2gCN5pvvCQ8, Piece ID: 3DFZTINQGUUTX3YDEXERPDRZNN5CKAWOMVAQHL23ES2U5YHH2RAA) context deadline exceeded; piecestore: (Node ID: 1X5A7bLqMEQuaMbf9xZ7PDUD4161sz2QqdM9r59y3bsyPBJhaK, Piece ID: 65WTA5P6FPBVAHUJEAX3TIFC23M3A52IZ6FLARKO5RPSSIBZEXWQ) context deadline exceeded ======LOGS====== 38:4484 , piece_id=4US2ZAFTL4NOMWPTAV6ABOM4GE673PJHVJ7C5B3YGYDVFQUPQLXA, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.326171965 (524h44m33.197255s) [noise=true , address=77.90.31.12:4603 , piece_id=Y2CDTLJ5S6DCNVTSXNOYT6FU7TSJYREG2BTIDPMOZAQ6FDXHQYAA, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.326204638 (524h44m33.197287683s) [noise=true , address=77.90.31.13:4644 , piece_id=EP2667PIQIHUWPTWD4REAWTQFSC3VR7V7VZQ344XJH4ASSJP4EMA, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.326588799 (524h44m33.197671844s) [noise=true , address=145.79.227.44:4213 , piece_id=LE27IRRSIIE7T7JQLEMMW4DNJ3XMSRFOPE2DF2XV5XCNWZU3HDEQ, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.326927453 (524h44m33.198010498s) [noise=true , address=77.90.31.15:4721 , piece_id=3A72PLJ5CH6JGOMDL4PUYK3DVIUXRDHVQQA3WA2RW77FXKY46YOQ, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.326992878 (524h44m33.198075923s) [noise=true , address=145.79.227.32:4860 , piece_id=DXII7EZHDNAX6XTRBF6T2N2LTNP5H3DPHQEGDWKN7ZL7V3MVLDXQ, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.327699914 (524h44m33.198782949s) [noise=true , address=45.91.116.12:4258 , piece_id=3DFZTINQGUUTX3YDEXERPDRZNN5CKAWOMVAQHL23ES2U5YHH2RAA, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.327720574 (524h44m33.198803609s) [noise=true , address=145.79.227.34:4914 , piece_id=J7JR3XUPHJI3K4TZ23NECWAGL6YLFURO6MB7QR64L7LCEIBMWUXA, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.328318352 (524h44m33.199401387s) [noise=true , address=77.90.31.16:4746 , piece_id=UJJK5KRSEB3JCBXVUSE4TDHJ5JMIEDIGT4EWETKJCBEE46HSM2VA, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.328412561 (524h44m33.199495616s) [noise=true , address=145.79.227.30:4775 , piece_id=D7SJXV4WU5ZGJVYZHZ2UOVAF72ANYWSOCS5BH5U3Z3EI7MXW536A, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.328792936 (524h44m33.199875981s) [noise=true , address=77.90.31.11:4558 , piece_id=HGCZHZGYN6OPOFJJEIMQG2LXMGA6JDHIDDN3RTRF7XSZ56OL3M7Q, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.329056156 (524h44m33.200139191s) [noise=true , address=77.90.31.21:4905 , piece_id=65WTA5P6FPBVAHUJEAX3TIFC23M3A52IZ6FLARKO5RPSSIBZEXWQ, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.329239847 (524h44m33.200322892s) [noise=true , address=64.49.13.169:4709 , piece_id=XOM2255ZNKHB3I26XEYPYVMPCX4QTYGKL6NLXQOPOZ4PQCELVEAQ, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.329737634 (524h44m33.20082068s) [noise=true , address=145.79.227.40:4072 , piece_id=56RK7LN3GVLMGIYZD6FQ2FU23ADTGNWUYMJUMIHH5HHKATLYOQWQ, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.330431346 (524h44m33.201514391s) [noise=true , address=45.91.116.9:4287 , piece_id=F3TDAQGQWDLWZL2WSKWKMMB77TCNBHLIZVY6NPM5MX2ARUSVW37Q, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:34.611654557 (524h44m34.482737602s) [upload=460306]: Next returned nil segment Dec 27 12:13:34.611669636 (524h44m34.482752681s) [upload=460306]: Waiting for error group managing segments... Dec 27 12:13:34.940178727 (524h44m34.811261762s) [noise=true , address=64.49.13.138:4484 , piece_id=4US2ZAFTL4NOMWPTAV6ABOM4GE673PJHVJ7C5B3YGYDVFQUPQLXA, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.218550166 (524h44m35.089633211s) [noise=true , address=145.79.227.32:4860 , piece_id=DXII7EZHDNAX6XTRBF6T2N2LTNP5H3DPHQEGDWKN7ZL7V3MVLDXQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.226012146 (524h44m35.097095191s) [noise=true , address=145.79.227.34:4914 , piece_id=J7JR3XUPHJI3K4TZ23NECWAGL6YLFURO6MB7QR64L7LCEIBMWUXA, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.400566879 (524h44m35.271649914s) [noise=true , address=145.79.227.33:4891 , piece_id=2R47TJ3SLIMPLH4JZTSINHBZXQ3JJKXVETIMILFATRXEI6DZLJLQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.404347268 (524h44m35.275430303s) [noise=true , address=77.90.31.21:4905 , piece_id=65WTA5P6FPBVAHUJEAX3TIFC23M3A52IZ6FLARKO5RPSSIBZEXWQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.416074639 (524h44m35.287157674s) [noise=true , address=77.90.31.13:4644 , piece_id=EP2667PIQIHUWPTWD4REAWTQFSC3VR7V7VZQ344XJH4ASSJP4EMA, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.452194618 (524h44m35.323277663s) [noise=true , address=45.91.116.9:4287 , piece_id=F3TDAQGQWDLWZL2WSKWKMMB77TCNBHLIZVY6NPM5MX2ARUSVW37Q, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.525559618 (524h44m35.396642663s) [noise=true , address=45.91.116.12:4258 , piece_id=3DFZTINQGUUTX3YDEXERPDRZNN5CKAWOMVAQHL23ES2U5YHH2RAA, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.710902755 (524h44m35.58198579s) [noise=true , address=77.90.31.12:4603 , piece_id=Y2CDTLJ5S6DCNVTSXNOYT6FU7TSJYREG2BTIDPMOZAQ6FDXHQYAA, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.833345128 (524h44m35.704428173s) [noise=true , address=77.90.31.15:4721 , piece_id=3A72PLJ5CH6JGOMDL4PUYK3DVIUXRDHVQQA3WA2RW77FXKY46YOQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:36.102928202 (524h44m35.974011237s) [noise=true , address=145.79.227.44:4213 , piece_id=LE27IRRSIIE7T7JQLEMMW4DNJ3XMSRFOPE2DF2XV5XCNWZU3HDEQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:36.153147128 (524h44m36.024230183s) [noise=true , address=145.79.227.40:4072 , piece_id=56RK7LN3GVLMGIYZD6FQ2FU23ADTGNWUYMJUMIHH5HHKATLYOQWQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:36.208908849 (524h44m36.079991884s) [noise=true , address=145.79.227.30:4775 , piece_id=D7SJXV4WU5ZGJVYZHZ2UOVAF72ANYWSOCS5BH5U3Z3EI7MXW536A, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:36.410529180 (524h44m36.281612215s) [noise=true , address=45.91.116.12:4248 , piece_id=G5SOGWARFYFBXQRSRB5OXZZEKN6BMGN3PQC2RBB7JHN55QUPRXIQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:36.410547656 (524h44m36.281630691s) [seg_pos={0 0}, upload=460306]: Segment reached optimal threshold of 14 pieces. Dec 27 12:13:36.410724162 (524h44m36.281807197s) [noise=true , address=77.90.31.11:4558 , piece_id=HGCZHZGYN6OPOFJJEIMQG2LXMGA6JDHIDDN3RTRF7XSZ56OL3M7Q, seg_pos={0 0}, upload=460306]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mGEP1n1icC2gEkyRVy6BeM4vsrgqYWaQeB5rUtxjrWQjyFhTS): protocol: expected piece hash; context canceled; EOF Dec 27 12:13:36.410730103 (524h44m36.281813138s) [noise=true , address=77.90.31.11:4558 , piece_id=HGCZHZGYN6OPOFJJEIMQG2LXMGA6JDHIDDN3RTRF7XSZ56OL3M7Q, seg_pos={0 0}, upload=460306]: Long tail cancellation due to optimal threshold Dec 27 12:13:36.410916448 (524h44m36.281999483s) [noise=true , address=77.90.31.16:4746 , piece_id=UJJK5KRSEB3JCBXVUSE4TDHJ5JMIEDIGT4EWETKJCBEE46HSM2VA, seg_pos={0 0}, upload=460306]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o): protocol: write tcp 188.166.225.170:42390->77.90.31.16:4746: use of closed network connection; write tcp 188.166.225.170:42390->77.90.31.16:4746: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42390->77.90.31.16:4746: use of closed network connection Dec 27 12:13:36.410922750 (524h44m36.282005785s) [noise=true , address=77.90.31.16:4746 , piece_id=UJJK5KRSEB3JCBXVUSE4TDHJ5JMIEDIGT4EWETKJCBEE46HSM2VA, seg_pos={0 0}, upload=460306]: Long tail cancellation due to optimal threshold Dec 27 12:13:36.411020866 (524h44m36.282103911s) [noise=true , address=193.149.164.143:4699 , piece_id=4A5T3OSNSHNKNXKRWOWVJC222TNAISNRGN3NCXQGVFKNHVDBKXHQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LncNSBhhreg9JTByJPSB7eGUEu88Eyd3jc9pg24VsEjp4FULq): protocol: expected piece hash; context canceled; EOF Dec 27 12:13:36.411036306 (524h44m36.282119341s) [noise=true , address=193.149.164.143:4699 , piece_id=4A5T3OSNSHNKNXKRWOWVJC222TNAISNRGN3NCXQGVFKNHVDBKXHQ, seg_pos={0 0}, upload=460306]: Long tail cancellation due to optimal threshold Dec 27 12:13:36.411086592 (524h44m36.282169637s) [noise=true , address=64.49.13.169:4709 , piece_id=XOM2255ZNKHB3I26XEYPYVMPCX4QTYGKL6NLXQOPOZ4PQCELVEAQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:58550->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:58550->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58550->64.49.13.169:4709: use of closed network connection Dec 27 12:13:36.411093095 (524h44m36.28217613s) [noise=true , address=64.49.13.169:4709 , piece_id=XOM2255ZNKHB3I26XEYPYVMPCX4QTYGKL6NLXQOPOZ4PQCELVEAQ, seg_pos={0 0}, upload=460306]: Long tail cancellation due to optimal threshold Dec 27 12:13:36.411118663 (524h44m36.282201708s) [seg_pos={0 0}, upload=460306]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 12:13:36.411202012 (524h44m36.282285047s) [upload=460306]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 12:13:36.411207102 (524h44m36.282290137s) [upload=460306]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 12:13:36.701956341 (524h44m36.573039386s) [upload=460306]: Done uploading segments... Dec 27 12:13:36.701976790 (524h44m36.573059825s) [upload=460306]: Upload finished. err: <nil> Duration: 49.792832524s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [...XX.X....XX......X.X.X.XX.X....X.X..X..XXXXXXX.X.XX...XX.X] duration: 30.003152718s trace: https://jaeger.dev.storj.io/trace/6cf6706bb6e7ded8 FAIL (2025-12-27 12:24:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: NW63JDCXYKJAS7F6Z7M5P75VJXLXJF6YAKEWD4OCBQEQSO6LXJTQ) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: G5F6ZX6LUCOA2B3XTWY7GDKHO5KF5GRI6U26J3VF2S76A25SQ62Q) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: F43KDFWHQLIW62AKO35HBR7GLDH37I3PVM6KCWPRS6AYFGTWSOEA) context deadline exceeded ======LOGS====== Dec 27 12:23:51.817164147 (524h54m51.688247182s) [upload=460449]: Starting upload Dec 27 12:23:51.817252766 (524h54m51.688335841s) [upload=460449]: Done starting upload Dec 27 12:23:51.817304173 (524h54m51.688387208s) [upload=460449]: Uploading segments... Dec 27 12:23:51.817519284 (524h54m51.688602329s) [upload=460449]: Got next segment. Inline: false Dec 27 12:23:51.817551424 (524h54m51.688634459s) [upload=460449]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 12:23:51.817556605 (524h54m51.68863964s) [upload=460449]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 12:23:52.067545508 (524h54m51.938628543s) [seg_pos={0 0}, upload=460449]: Begin upload segment... Dec 27 12:23:52.067599912 (524h54m51.938682947s) [seg_pos={0 0}, upload=460449]: Done begin upload segment. Dec 27 12:23:52.067630540 (524h54m51.938713575s) [noise=true , address=145.79.227.46:4603 , piece_id=C2P6AAAZLZTHHV27KWBX7Q2CMAMGCSIY7DSHELEM66YAEQTUZXGA, seg_pos={0 0}, upload=460449]: Uploading piece... Dec 27 12:23:52.067659305 (524h54m51.93874234s) [noise=true , address=45.91.116.11:4118 , piece_id=NW63JDCXYKJAS7F6Z7M5P75VJXLXJF6YAKEWD4OCBQEQSO6LXJTQ, seg_pos={0 0}, upload=460449]: Uploading piece... Dec 27 12:23:52.068259889 (524h54m51.939342924s) [noise=true , address=145.79.227.11:4730 , piece_id=OYQQAZVGMZ4UMBCEMFVA4TCC3MYPZ7V66QJNPQTDLZKL2NPTGJXA, seg_pos={0 0}, upload=460449]: Uploading piece... Dec 27 12:23:52.068582152 (524h54m51.939665187s) [noise=true , address=145.79.227.47:4265 , piece_id=JGO5ULVAKZQNV6SISXFVVYZEEI3ISZ6ZJITH3CE3G5QBYJ3QHKRQ, seg_pos={0 0}, upload=460449]: Uploading piece... Dec 27 12:23:52.068880159 (524h54m51.939963194s) [noise=true , address=45.91.116.12:4227 , piece_id=SJZW2EZSFQFMREN2CPBTZNB5EU5LFU2SYBE2HPSLPE5WORVNKN5A, seg_pos={0 0}, upload=460449]: Uploading piece... Dec 27 12:23:52.070426054 (524h54m51.941509099s) [noise=true , address=45.91.116.10:4082 , piece_id=EHOHDYBYMBL5LL3YGW65XWMHZREQWFWQ7BKIWIPJAK275MOL7XWQ, seg_pos={0 0}, upload=460449]: Uploading piece... Dec 27 12:23:52.070802531 (524h54m51.941885566s) [noise=true , address=145.79.227.14:4837 , piece_id=RIIIVSMSAWEUWIGRHZYG2IIP5OJ6P4GRAZBZDT36A7Q7BKGE7DGA, seg_pos={0 0}, upload=460449]: Uploading piece... Dec 27 12:23:52.071178927 (524h54m51.942261962s) [noise=true , address=145.79.227.25:4617 , piece_id=DXHLYI5LJNAK2UASJAMY4CHJI6X43NJCXMGP53ZROCHQV33RIU5Q, seg_pos={0 0}, upload=460449]: Uploading piece... Dec 27 12:23:52.072786448 (524h54m51.943869483s) [noise=true , address=45.91.116.14:4189 , piece_id=I44OLDAZY6NLSYVZB233KVXOITGS2RXCFNVAUENGLUOQXBERAGUA, seg_pos={0 0}, upload=460449]: Uploading piece... Dec 27 12:23:52.074481455 (524h54m51.9455645s) [noise=true , address=145.79.227.2:4113 , piece_id=BY47A2PLISIYPP3SBPNLSD2SYABHSTEZQZHHXOXXFXDHFKPY66LA, seg_pos={0 0}, upload=460449]: Uploading piece... Dec 27 12:23:52.078972246 (524h54m51.950055282s) [noise=true , address=145.79.227.39:4015 , piece_id=665EBMIGZKO755SVRAA6GG4GO3JWX62QAJBVKIUQILFY37XN5OEQ, seg_pos={0 0}, upload=460449]: Uploading piece... Dec 27 12:23:52.081374140 (524h54m51.952457175s) [noise=true , address=145.79.227.42:4122 , piece_id=SSIQ3D3UAH5R6VYBUE5JKCARE3RMIUGJ2HC4AJF2BDKKDDCGVPIA, seg_pos={0 0}, upload=460449]: Uploading piece... Dec 27 12:23:52.082854279 (524h54m51.953937314s) [noise=true , address=45.91.116.9:4264 , piece_id=G5F6ZX6LUCOA2B3XTWY7GDKHO5KF5GRI6U26J3VF2S76A25SQ62Q, seg_pos={0 0}, upload=460449]: Uploading piece... Dec 27 12:23:52.083655705 (524h54m51.95473874s) [noise=true , address=145.79.227.7:4295 , piece_id=ZMJD5AAMLLJAL622YTPCEGDRFUHOT5GM4WCWJRLIK7JYMOYNGCPA, seg_pos={0 0}, upload=460449]: Uploading piece... Dec 27 12:23:52.085709496 (524h54m51.956792531s) [noise=true , address=145.79.227.51:4409 , piece_id=4FLC4A3ON6QQCVNQ3DQUB35LL7IVOY6PT7JHOWFBKTETD6W6IUOA, seg_pos={0 0}, upload=460449]: Uploading piece... Dec 27 12:23:52.089855871 (524h54m51.960938916s) [noise=true , address=45.91.116.8:4048 , piece_id=F43KDFWHQLIW62AKO35HBR7GLDH37I3PVM6KCWPRS6AYFGTWSOEA, seg_pos={0 0}, upload=460449]: Uploading piece... Dec 27 12:23:53.791147820 (524h54m53.662230865s) [upload=460449]: Next returned nil segment Dec 27 12:23:53.791162068 (524h54m53.662245113s) [upload=460449]: Waiting for error group managing segments... Dec 27 12:23:54.083030148 (524h54m53.954113193s) [noise=true , address=145.79.227.42:4122 , piece_id=SSIQ3D3UAH5R6VYBUE5JKCARE3RMIUGJ2HC4AJF2BDKKDDCGVPIA, seg_pos={0 0}, upload=460449]: Done uploading piece. err: <nil> Dec 27 12:23:54.085281024 (524h54m53.956364069s) [noise=true , address=145.79.227.2:4113 , piece_id=BY47A2PLISIYPP3SBPNLSD2SYABHSTEZQZHHXOXXFXDHFKPY66LA, seg_pos={0 0}, upload=460449]: Done uploading piece. err: <nil> Dec 27 12:23:54.086108228 (524h54m53.957191273s) [noise=true , address=145.79.227.47:4265 , piece_id=JGO5ULVAKZQNV6SISXFVVYZEEI3ISZ6ZJITH3CE3G5QBYJ3QHKRQ, seg_pos={0 0}, upload=460449]: Done uploading piece. err: <nil> Dec 27 12:23:54.089593665 (524h54m53.96067671s) [noise=true , address=145.79.227.7:4295 , piece_id=ZMJD5AAMLLJAL622YTPCEGDRFUHOT5GM4WCWJRLIK7JYMOYNGCPA, seg_pos={0 0}, upload=460449]: Done uploading piece. err: <nil> Dec 27 12:23:54.098052552 (524h54m53.969135587s) [noise=true , address=145.79.227.14:4837 , piece_id=RIIIVSMSAWEUWIGRHZYG2IIP5OJ6P4GRAZBZDT36A7Q7BKGE7DGA, seg_pos={0 0}, upload=460449]: Done uploading piece. err: <nil> Dec 27 12:23:54.102026420 (524h54m53.973109455s) [noise=true , address=145.79.227.46:4603 , piece_id=C2P6AAAZLZTHHV27KWBX7Q2CMAMGCSIY7DSHELEM66YAEQTUZXGA, seg_pos={0 0}, upload=460449]: Done uploading piece. err: <nil> Dec 27 12:23:54.102841882 (524h54m53.973924947s) [noise=true , address=145.79.227.25:4617 , piece_id=DXHLYI5LJNAK2UASJAMY4CHJI6X43NJCXMGP53ZROCHQV33RIU5Q, seg_pos={0 0}, upload=460449]: Done uploading piece. err: <nil> Dec 27 12:23:54.109331379 (524h54m53.980414424s) [noise=true , address=145.79.227.51:4409 , piece_id=4FLC4A3ON6QQCVNQ3DQUB35LL7IVOY6PT7JHOWFBKTETD6W6IUOA, seg_pos={0 0}, upload=460449]: Done uploading piece. err: <nil> Dec 27 12:23:54.111015927 (524h54m53.982098972s) [noise=true , address=145.79.227.39:4015 , piece_id=665EBMIGZKO755SVRAA6GG4GO3JWX62QAJBVKIUQILFY37XN5OEQ, seg_pos={0 0}, upload=460449]: Done uploading piece. err: <nil> Dec 27 12:23:54.111572667 (524h54m53.982655703s) [noise=true , address=145.79.227.11:4730 , piece_id=OYQQAZVGMZ4UMBCEMFVA4TCC3MYPZ7V66QJNPQTDLZKL2NPTGJXA, seg_pos={0 0}, upload=460449]: Done uploading piece. err: <nil> Dec 27 12:23:54.326498475 (524h54m54.19758152s) [noise=true , address=45.91.116.11:4118 , piece_id=NW63JDCXYKJAS7F6Z7M5P75VJXLXJF6YAKEWD4OCBQEQSO6LXJTQ, seg_pos={0 0}, upload=460449]: Done uploading piece. err: <nil> Dec 27 12:23:54.432110971 (524h54m54.303194016s) [noise=true , address=45.91.116.9:4264 , piece_id=G5F6ZX6LUCOA2B3XTWY7GDKHO5KF5GRI6U26J3VF2S76A25SQ62Q, seg_pos={0 0}, upload=460449]: Done uploading piece. err: <nil> Dec 27 12:23:54.726187126 (524h54m54.597270171s) [noise=true , address=45.91.116.8:4048 , piece_id=F43KDFWHQLIW62AKO35HBR7GLDH37I3PVM6KCWPRS6AYFGTWSOEA, seg_pos={0 0}, upload=460449]: Done uploading piece. err: <nil> Dec 27 12:23:54.726218175 (524h54m54.59730122s) [seg_pos={0 0}, upload=460449]: Segment reached optimal threshold of 13 pieces. Dec 27 12:23:54.726598468 (524h54m54.597681513s) [noise=true , address=45.91.116.14:4189 , piece_id=I44OLDAZY6NLSYVZB233KVXOITGS2RXCFNVAUENGLUOQXBERAGUA, seg_pos={0 0}, upload=460449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 27 12:23:54.726615822 (524h54m54.597698857s) [noise=true , address=45.91.116.14:4189 , piece_id=I44OLDAZY6NLSYVZB233KVXOITGS2RXCFNVAUENGLUOQXBERAGUA, seg_pos={0 0}, upload=460449]: Long tail cancellation due to optimal threshold Dec 27 12:23:54.726647993 (524h54m54.597731038s) [noise=true , address=45.91.116.10:4082 , piece_id=EHOHDYBYMBL5LL3YGW65XWMHZREQWFWQ7BKIWIPJAK275MOL7XWQ, seg_pos={0 0}, upload=460449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: read tcp 188.166.225.170:54420->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 12:23:54.726658943 (524h54m54.597741979s) [noise=true , address=45.91.116.10:4082 , piece_id=EHOHDYBYMBL5LL3YGW65XWMHZREQWFWQ7BKIWIPJAK275MOL7XWQ, seg_pos={0 0}, upload=460449]: Long tail cancellation due to optimal threshold Dec 27 12:23:54.726840459 (524h54m54.597923504s) [noise=true , address=45.91.116.12:4227 , piece_id=SJZW2EZSFQFMREN2CPBTZNB5EU5LFU2SYBE2HPSLPE5WORVNKN5A, seg_pos={0 0}, upload=460449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: read tcp 188.166.225.170:45966->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 12:23:54.726848325 (524h54m54.59793137s) [noise=true , address=45.91.116.12:4227 , piece_id=SJZW2EZSFQFMREN2CPBTZNB5EU5LFU2SYBE2HPSLPE5WORVNKN5A, seg_pos={0 0}, upload=460449]: Long tail cancellation due to optimal threshold Dec 27 12:23:54.726871178 (524h54m54.597954223s) [seg_pos={0 0}, upload=460449]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 12:23:54.726957032 (524h54m54.598040077s) [upload=460449]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 12:23:54.726963303 (524h54m54.598046348s) [upload=460449]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 12:23:55.033521536 (524h54m54.904604571s) [upload=460449]: Done uploading segments... Dec 27 12:23:55.033543488 (524h54m54.904626533s) [upload=460449]: Upload finished. err: <nil> Duration: 30.00317468s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 8.898555413s trace: https://jaeger.dev.storj.io/trace/578b4df2b1bacee FAIL (2025-12-27 12:24:21 UTC) orbiter warning: context deadline exceeded Duration: 8.898584879s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 2.450984729s trace: https://jaeger.dev.storj.io/trace/6dd289079d7aa430 FAIL (2025-12-27 12:24:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.451013254s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.X..........................................................] duration: 4.687168601s trace: https://jaeger.dev.storj.io/trace/12569321705fed16 FAIL (2025-12-27 12:23:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.687196043s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 6.276116995s trace: https://jaeger.dev.storj.io/trace/578b4df2b1bacee FAIL (2025-12-27 12:23:53 UTC) orbiter warning: context deadline exceeded Duration: 6.276143986s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 2.69733695s trace: https://jaeger.dev.storj.io/trace/6dd289079d7aa430 FAIL (2025-12-27 12:23:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.697439425s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................X.................X...........] duration: 49.688045155s trace: https://jaeger.dev.storj.io/trace/3b6d882384466243 FAIL (2025-12-27 12:23:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 13QCH91kRMQjVj2ASsVDcyXQ1aapQRVeUm6MMCGimsR6Ns973h, Piece ID: EP2667PIQIHUWPTWD4REAWTQFSC3VR7V7VZQ344XJH4ASSJP4EMA) context deadline exceeded; piecestore: (Node ID: 126Sou5pnMm2eE2FJpv6WYX2VSqJvZZf9rFFJD6hFYDetRD28zu, Piece ID: Y2CDTLJ5S6DCNVTSXNOYT6FU7TSJYREG2BTIDPMOZAQ6FDXHQYAA) context deadline exceeded; piecestore: (Node ID: 12eLKUg2vTnZMrLaraSPEoGXe3jDVBJYyURdcn4f2gCN5pvvCQ8, Piece ID: 3DFZTINQGUUTX3YDEXERPDRZNN5CKAWOMVAQHL23ES2U5YHH2RAA) context deadline exceeded; piecestore: (Node ID: 1X5A7bLqMEQuaMbf9xZ7PDUD4161sz2QqdM9r59y3bsyPBJhaK, Piece ID: 65WTA5P6FPBVAHUJEAX3TIFC23M3A52IZ6FLARKO5RPSSIBZEXWQ) context deadline exceeded ======LOGS====== 38:4484 , piece_id=4US2ZAFTL4NOMWPTAV6ABOM4GE673PJHVJ7C5B3YGYDVFQUPQLXA, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.326171965 (524h44m33.197255s) [noise=true , address=77.90.31.12:4603 , piece_id=Y2CDTLJ5S6DCNVTSXNOYT6FU7TSJYREG2BTIDPMOZAQ6FDXHQYAA, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.326204638 (524h44m33.197287683s) [noise=true , address=77.90.31.13:4644 , piece_id=EP2667PIQIHUWPTWD4REAWTQFSC3VR7V7VZQ344XJH4ASSJP4EMA, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.326588799 (524h44m33.197671844s) [noise=true , address=145.79.227.44:4213 , piece_id=LE27IRRSIIE7T7JQLEMMW4DNJ3XMSRFOPE2DF2XV5XCNWZU3HDEQ, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.326927453 (524h44m33.198010498s) [noise=true , address=77.90.31.15:4721 , piece_id=3A72PLJ5CH6JGOMDL4PUYK3DVIUXRDHVQQA3WA2RW77FXKY46YOQ, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.326992878 (524h44m33.198075923s) [noise=true , address=145.79.227.32:4860 , piece_id=DXII7EZHDNAX6XTRBF6T2N2LTNP5H3DPHQEGDWKN7ZL7V3MVLDXQ, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.327699914 (524h44m33.198782949s) [noise=true , address=45.91.116.12:4258 , piece_id=3DFZTINQGUUTX3YDEXERPDRZNN5CKAWOMVAQHL23ES2U5YHH2RAA, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.327720574 (524h44m33.198803609s) [noise=true , address=145.79.227.34:4914 , piece_id=J7JR3XUPHJI3K4TZ23NECWAGL6YLFURO6MB7QR64L7LCEIBMWUXA, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.328318352 (524h44m33.199401387s) [noise=true , address=77.90.31.16:4746 , piece_id=UJJK5KRSEB3JCBXVUSE4TDHJ5JMIEDIGT4EWETKJCBEE46HSM2VA, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.328412561 (524h44m33.199495616s) [noise=true , address=145.79.227.30:4775 , piece_id=D7SJXV4WU5ZGJVYZHZ2UOVAF72ANYWSOCS5BH5U3Z3EI7MXW536A, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.328792936 (524h44m33.199875981s) [noise=true , address=77.90.31.11:4558 , piece_id=HGCZHZGYN6OPOFJJEIMQG2LXMGA6JDHIDDN3RTRF7XSZ56OL3M7Q, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.329056156 (524h44m33.200139191s) [noise=true , address=77.90.31.21:4905 , piece_id=65WTA5P6FPBVAHUJEAX3TIFC23M3A52IZ6FLARKO5RPSSIBZEXWQ, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.329239847 (524h44m33.200322892s) [noise=true , address=64.49.13.169:4709 , piece_id=XOM2255ZNKHB3I26XEYPYVMPCX4QTYGKL6NLXQOPOZ4PQCELVEAQ, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.329737634 (524h44m33.20082068s) [noise=true , address=145.79.227.40:4072 , piece_id=56RK7LN3GVLMGIYZD6FQ2FU23ADTGNWUYMJUMIHH5HHKATLYOQWQ, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.330431346 (524h44m33.201514391s) [noise=true , address=45.91.116.9:4287 , piece_id=F3TDAQGQWDLWZL2WSKWKMMB77TCNBHLIZVY6NPM5MX2ARUSVW37Q, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:34.611654557 (524h44m34.482737602s) [upload=460306]: Next returned nil segment Dec 27 12:13:34.611669636 (524h44m34.482752681s) [upload=460306]: Waiting for error group managing segments... Dec 27 12:13:34.940178727 (524h44m34.811261762s) [noise=true , address=64.49.13.138:4484 , piece_id=4US2ZAFTL4NOMWPTAV6ABOM4GE673PJHVJ7C5B3YGYDVFQUPQLXA, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.218550166 (524h44m35.089633211s) [noise=true , address=145.79.227.32:4860 , piece_id=DXII7EZHDNAX6XTRBF6T2N2LTNP5H3DPHQEGDWKN7ZL7V3MVLDXQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.226012146 (524h44m35.097095191s) [noise=true , address=145.79.227.34:4914 , piece_id=J7JR3XUPHJI3K4TZ23NECWAGL6YLFURO6MB7QR64L7LCEIBMWUXA, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.400566879 (524h44m35.271649914s) [noise=true , address=145.79.227.33:4891 , piece_id=2R47TJ3SLIMPLH4JZTSINHBZXQ3JJKXVETIMILFATRXEI6DZLJLQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.404347268 (524h44m35.275430303s) [noise=true , address=77.90.31.21:4905 , piece_id=65WTA5P6FPBVAHUJEAX3TIFC23M3A52IZ6FLARKO5RPSSIBZEXWQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.416074639 (524h44m35.287157674s) [noise=true , address=77.90.31.13:4644 , piece_id=EP2667PIQIHUWPTWD4REAWTQFSC3VR7V7VZQ344XJH4ASSJP4EMA, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.452194618 (524h44m35.323277663s) [noise=true , address=45.91.116.9:4287 , piece_id=F3TDAQGQWDLWZL2WSKWKMMB77TCNBHLIZVY6NPM5MX2ARUSVW37Q, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.525559618 (524h44m35.396642663s) [noise=true , address=45.91.116.12:4258 , piece_id=3DFZTINQGUUTX3YDEXERPDRZNN5CKAWOMVAQHL23ES2U5YHH2RAA, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.710902755 (524h44m35.58198579s) [noise=true , address=77.90.31.12:4603 , piece_id=Y2CDTLJ5S6DCNVTSXNOYT6FU7TSJYREG2BTIDPMOZAQ6FDXHQYAA, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.833345128 (524h44m35.704428173s) [noise=true , address=77.90.31.15:4721 , piece_id=3A72PLJ5CH6JGOMDL4PUYK3DVIUXRDHVQQA3WA2RW77FXKY46YOQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:36.102928202 (524h44m35.974011237s) [noise=true , address=145.79.227.44:4213 , piece_id=LE27IRRSIIE7T7JQLEMMW4DNJ3XMSRFOPE2DF2XV5XCNWZU3HDEQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:36.153147128 (524h44m36.024230183s) [noise=true , address=145.79.227.40:4072 , piece_id=56RK7LN3GVLMGIYZD6FQ2FU23ADTGNWUYMJUMIHH5HHKATLYOQWQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:36.208908849 (524h44m36.079991884s) [noise=true , address=145.79.227.30:4775 , piece_id=D7SJXV4WU5ZGJVYZHZ2UOVAF72ANYWSOCS5BH5U3Z3EI7MXW536A, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:36.410529180 (524h44m36.281612215s) [noise=true , address=45.91.116.12:4248 , piece_id=G5SOGWARFYFBXQRSRB5OXZZEKN6BMGN3PQC2RBB7JHN55QUPRXIQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:36.410547656 (524h44m36.281630691s) [seg_pos={0 0}, upload=460306]: Segment reached optimal threshold of 14 pieces. Dec 27 12:13:36.410724162 (524h44m36.281807197s) [noise=true , address=77.90.31.11:4558 , piece_id=HGCZHZGYN6OPOFJJEIMQG2LXMGA6JDHIDDN3RTRF7XSZ56OL3M7Q, seg_pos={0 0}, upload=460306]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mGEP1n1icC2gEkyRVy6BeM4vsrgqYWaQeB5rUtxjrWQjyFhTS): protocol: expected piece hash; context canceled; EOF Dec 27 12:13:36.410730103 (524h44m36.281813138s) [noise=true , address=77.90.31.11:4558 , piece_id=HGCZHZGYN6OPOFJJEIMQG2LXMGA6JDHIDDN3RTRF7XSZ56OL3M7Q, seg_pos={0 0}, upload=460306]: Long tail cancellation due to optimal threshold Dec 27 12:13:36.410916448 (524h44m36.281999483s) [noise=true , address=77.90.31.16:4746 , piece_id=UJJK5KRSEB3JCBXVUSE4TDHJ5JMIEDIGT4EWETKJCBEE46HSM2VA, seg_pos={0 0}, upload=460306]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o): protocol: write tcp 188.166.225.170:42390->77.90.31.16:4746: use of closed network connection; write tcp 188.166.225.170:42390->77.90.31.16:4746: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42390->77.90.31.16:4746: use of closed network connection Dec 27 12:13:36.410922750 (524h44m36.282005785s) [noise=true , address=77.90.31.16:4746 , piece_id=UJJK5KRSEB3JCBXVUSE4TDHJ5JMIEDIGT4EWETKJCBEE46HSM2VA, seg_pos={0 0}, upload=460306]: Long tail cancellation due to optimal threshold Dec 27 12:13:36.411020866 (524h44m36.282103911s) [noise=true , address=193.149.164.143:4699 , piece_id=4A5T3OSNSHNKNXKRWOWVJC222TNAISNRGN3NCXQGVFKNHVDBKXHQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LncNSBhhreg9JTByJPSB7eGUEu88Eyd3jc9pg24VsEjp4FULq): protocol: expected piece hash; context canceled; EOF Dec 27 12:13:36.411036306 (524h44m36.282119341s) [noise=true , address=193.149.164.143:4699 , piece_id=4A5T3OSNSHNKNXKRWOWVJC222TNAISNRGN3NCXQGVFKNHVDBKXHQ, seg_pos={0 0}, upload=460306]: Long tail cancellation due to optimal threshold Dec 27 12:13:36.411086592 (524h44m36.282169637s) [noise=true , address=64.49.13.169:4709 , piece_id=XOM2255ZNKHB3I26XEYPYVMPCX4QTYGKL6NLXQOPOZ4PQCELVEAQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:58550->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:58550->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58550->64.49.13.169:4709: use of closed network connection Dec 27 12:13:36.411093095 (524h44m36.28217613s) [noise=true , address=64.49.13.169:4709 , piece_id=XOM2255ZNKHB3I26XEYPYVMPCX4QTYGKL6NLXQOPOZ4PQCELVEAQ, seg_pos={0 0}, upload=460306]: Long tail cancellation due to optimal threshold Dec 27 12:13:36.411118663 (524h44m36.282201708s) [seg_pos={0 0}, upload=460306]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 12:13:36.411202012 (524h44m36.282285047s) [upload=460306]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 12:13:36.411207102 (524h44m36.282290137s) [upload=460306]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 12:13:36.701956341 (524h44m36.573039386s) [upload=460306]: Done uploading segments... Dec 27 12:13:36.701976790 (524h44m36.573059825s) [upload=460306]: Upload finished. err: <nil> Duration: 49.688059834s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 6.472652587s trace: https://jaeger.dev.storj.io/trace/578b4df2b1bacee FAIL (2025-12-27 12:23:28 UTC) orbiter warning: context deadline exceeded Duration: 6.472687453s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..X.........................................................] duration: 3.590234235s trace: https://jaeger.dev.storj.io/trace/12569321705fed16 FAIL (2025-12-27 12:23:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.590266977s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [....XX.X....XX......X.X.X.XX.X....X.X..X..XXXXXXX.X.XX...XX.] duration: 6.075541374s trace: https://jaeger.dev.storj.io/trace/3085452edd720d63 FAIL (2025-12-27 12:22:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: TNYFLGHLEPYUXKB4UJ4GL2SSQSEZFEL3IZ3NVB6A2XYRZDAAVBWQ) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: 3BGF2ARRRGA4WZGFVBQQGVEMYUGC66TGQ65Q5SIONKINMNNOCVIA) context deadline exceeded ======LOGS====== Dec 27 12:22:09.025089235 (524h53m8.89617227s) [upload=460428]: Starting upload Dec 27 12:22:09.025170170 (524h53m8.896253215s) [upload=460428]: Done starting upload Dec 27 12:22:09.025359219 (524h53m8.896442265s) [upload=460428]: Uploading segments... Dec 27 12:22:09.025394326 (524h53m8.896477371s) [upload=460428]: Got next segment. Inline: false Dec 27 12:22:09.025418753 (524h53m8.896501798s) [upload=460428]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 12:22:09.025423201 (524h53m8.896506237s) [upload=460428]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 12:22:09.268893583 (524h53m9.139976628s) [seg_pos={0 0}, upload=460428]: Begin upload segment... Dec 27 12:22:09.268944200 (524h53m9.140027235s) [seg_pos={0 0}, upload=460428]: Done begin upload segment. Dec 27 12:22:09.269000027 (524h53m9.140083062s) [noise=true , address=145.79.227.35:4481 , piece_id=BDOVCOR7TNCF4XKG2DKNOCHCKQ6535MR3B7GDFMTVZDAITPUK5GA, seg_pos={0 0}, upload=460428]: Uploading piece... Dec 27 12:22:09.269027208 (524h53m9.140110243s) [noise=true , address=145.79.227.36:4519 , piece_id=XSNAEIAUAGSJLUQ7E2N3CEW3Z6NRKYEXM3IOBJWDYPUM4IKWFIQQ, seg_pos={0 0}, upload=460428]: Uploading piece... Dec 27 12:22:09.269609176 (524h53m9.140692211s) [noise=true , address=45.91.116.10:4084 , piece_id=3BGF2ARRRGA4WZGFVBQQGVEMYUGC66TGQ65Q5SIONKINMNNOCVIA, seg_pos={0 0}, upload=460428]: Uploading piece... Dec 27 12:22:09.269984872 (524h53m9.141067907s) [noise=true , address=77.90.31.11:4553 , piece_id=IEDCEK4LCDFX7ODJL2RIPT2F7ZOK3R4DCRUKPH7C6FULCGHCWTLA, seg_pos={0 0}, upload=460428]: Uploading piece... Dec 27 12:22:09.270586687 (524h53m9.141669722s) [noise=true , address=45.91.116.14:4192 , piece_id=NXXJWPGS3Y72KS53TIRJI444Q74O6USMQMQ6MMF4HYBOTYUOYVCA, seg_pos={0 0}, upload=460428]: Uploading piece... Dec 27 12:22:09.271050620 (524h53m9.142133665s) [noise=true , address=77.90.31.21:4880 , piece_id=URUDPQFHT7SZBWVLRJXTL7EITRJF2C7BMNX6K5U4WBCIOVRUSDBQ, seg_pos={0 0}, upload=460428]: Uploading piece... Dec 27 12:22:09.271699396 (524h53m9.142782431s) [noise=true , address=145.79.227.4:4185 , piece_id=EUE6RNGIPQ2HAX6CBCI55ANFIA3ITZB4T4PTZ7SNM2KEMQWN47IQ, seg_pos={0 0}, upload=460428]: Uploading piece... Dec 27 12:22:09.273021724 (524h53m9.144104759s) [noise=true , address=145.79.227.45:4229 , piece_id=CNTCTWZPN7CWXUUPSQ2S2I2POUUS3AJWH47ZJ6LRE3HI7QATWGMA, seg_pos={0 0}, upload=460428]: Uploading piece... Dec 27 12:22:09.273296688 (524h53m9.144379723s) [noise=true , address=145.79.227.14:4837 , piece_id=E5G54MM2XSSNWXJPMWJIEZAKXB6ZSFU6IVUBZQOJH6SAGYPJDWLA, seg_pos={0 0}, upload=460428]: Uploading piece... Dec 27 12:22:09.274364952 (524h53m9.145447997s) [noise=true , address=145.79.227.10:4402 , piece_id=YTRTL32F5NBVGJLK47LR2VF3V7MP6LNOCFIIMEAV27TZ2WJYDBKA, seg_pos={0 0}, upload=460428]: Uploading piece... Dec 27 12:22:09.274771416 (524h53m9.145854451s) [noise=true , address=145.79.227.42:4123 , piece_id=3IZULC56BEE75VT66DSM5HCQN5R6H2LS2Y627VJHPBCOO5WWUTOA, seg_pos={0 0}, upload=460428]: Uploading piece... Dec 27 12:22:09.276445052 (524h53m9.147528097s) [noise=true , address=145.79.227.25:4617 , piece_id=4S2ISYTU55S4NEGPOEBYAVAIT5GBD47HQ57226QPTGGHJODVTCYQ, seg_pos={0 0}, upload=460428]: Uploading piece... Dec 27 12:22:09.278172502 (524h53m9.149255547s) [noise=true , address=45.91.116.12:4227 , piece_id=TNYFLGHLEPYUXKB4UJ4GL2SSQSEZFEL3IZ3NVB6A2XYRZDAAVBWQ, seg_pos={0 0}, upload=460428]: Uploading piece... Dec 27 12:22:09.278988586 (524h53m9.150071631s) [noise=true , address=145.79.227.5:4221 , piece_id=FOV2F75NL4HXMPXGKSV5YKTYAANYUMEO7HDM2GLAZ5AT6WD6M6SA, seg_pos={0 0}, upload=460428]: Uploading piece... Dec 27 12:22:09.287289603 (524h53m9.158372638s) [noise=true , address=145.79.227.37:4567 , piece_id=PT3AQZ47CLHEHB3SOQFISBAJNGCGPNR2ZVA7GO53KAQH6A4H5QFA, seg_pos={0 0}, upload=460428]: Uploading piece... Dec 27 12:22:09.292163203 (524h53m9.163246258s) [noise=true , address=145.79.227.2:4113 , piece_id=RDT6KTALFOHFEKCLIW2YTUF3ZTM3KBYTVHXVII3PW7QVJOSLZRWA, seg_pos={0 0}, upload=460428]: Uploading piece... Dec 27 12:22:11.001147820 (524h53m10.872230875s) [upload=460428]: Next returned nil segment Dec 27 12:22:11.001168629 (524h53m10.872251674s) [upload=460428]: Waiting for error group managing segments... Dec 27 12:22:11.290317381 (524h53m11.161400446s) [noise=true , address=145.79.227.45:4229 , piece_id=CNTCTWZPN7CWXUUPSQ2S2I2POUUS3AJWH47ZJ6LRE3HI7QATWGMA, seg_pos={0 0}, upload=460428]: Done uploading piece. err: <nil> Dec 27 12:22:11.294623030 (524h53m11.165706085s) [noise=true , address=145.79.227.35:4481 , piece_id=BDOVCOR7TNCF4XKG2DKNOCHCKQ6535MR3B7GDFMTVZDAITPUK5GA, seg_pos={0 0}, upload=460428]: Done uploading piece. err: <nil> Dec 27 12:22:11.294785650 (524h53m11.165868695s) [noise=true , address=145.79.227.42:4123 , piece_id=3IZULC56BEE75VT66DSM5HCQN5R6H2LS2Y627VJHPBCOO5WWUTOA, seg_pos={0 0}, upload=460428]: Done uploading piece. err: <nil> Dec 27 12:22:11.295115197 (524h53m11.166198232s) [noise=true , address=145.79.227.10:4402 , piece_id=YTRTL32F5NBVGJLK47LR2VF3V7MP6LNOCFIIMEAV27TZ2WJYDBKA, seg_pos={0 0}, upload=460428]: Done uploading piece. err: <nil> Dec 27 12:22:11.295306602 (524h53m11.166389647s) [noise=true , address=145.79.227.25:4617 , piece_id=4S2ISYTU55S4NEGPOEBYAVAIT5GBD47HQ57226QPTGGHJODVTCYQ, seg_pos={0 0}, upload=460428]: Done uploading piece. err: <nil> Dec 27 12:22:11.295899040 (524h53m11.166982085s) [noise=true , address=145.79.227.2:4113 , piece_id=RDT6KTALFOHFEKCLIW2YTUF3ZTM3KBYTVHXVII3PW7QVJOSLZRWA, seg_pos={0 0}, upload=460428]: Done uploading piece. err: <nil> Dec 27 12:22:11.311065399 (524h53m11.182148444s) [noise=true , address=145.79.227.36:4519 , piece_id=XSNAEIAUAGSJLUQ7E2N3CEW3Z6NRKYEXM3IOBJWDYPUM4IKWFIQQ, seg_pos={0 0}, upload=460428]: Done uploading piece. err: <nil> Dec 27 12:22:11.311517950 (524h53m11.182600985s) [noise=true , address=145.79.227.4:4185 , piece_id=EUE6RNGIPQ2HAX6CBCI55ANFIA3ITZB4T4PTZ7SNM2KEMQWN47IQ, seg_pos={0 0}, upload=460428]: Done uploading piece. err: <nil> Dec 27 12:22:11.313099943 (524h53m11.184182988s) [noise=true , address=145.79.227.5:4221 , piece_id=FOV2F75NL4HXMPXGKSV5YKTYAANYUMEO7HDM2GLAZ5AT6WD6M6SA, seg_pos={0 0}, upload=460428]: Done uploading piece. err: <nil> Dec 27 12:22:11.318998825 (524h53m11.19008186s) [noise=true , address=145.79.227.14:4837 , piece_id=E5G54MM2XSSNWXJPMWJIEZAKXB6ZSFU6IVUBZQOJH6SAGYPJDWLA, seg_pos={0 0}, upload=460428]: Done uploading piece. err: <nil> Dec 27 12:22:11.323220775 (524h53m11.194303851s) [noise=true , address=145.79.227.37:4567 , piece_id=PT3AQZ47CLHEHB3SOQFISBAJNGCGPNR2ZVA7GO53KAQH6A4H5QFA, seg_pos={0 0}, upload=460428]: Done uploading piece. err: <nil> Dec 27 12:22:11.498870103 (524h53m11.369953148s) [noise=true , address=45.91.116.10:4084 , piece_id=3BGF2ARRRGA4WZGFVBQQGVEMYUGC66TGQ65Q5SIONKINMNNOCVIA, seg_pos={0 0}, upload=460428]: Done uploading piece. err: <nil> Dec 27 12:22:11.525496124 (524h53m11.396579159s) [noise=true , address=45.91.116.12:4227 , piece_id=TNYFLGHLEPYUXKB4UJ4GL2SSQSEZFEL3IZ3NVB6A2XYRZDAAVBWQ, seg_pos={0 0}, upload=460428]: Done uploading piece. err: <nil> Dec 27 12:22:11.525519288 (524h53m11.396602333s) [seg_pos={0 0}, upload=460428]: Segment reached optimal threshold of 13 pieces. Dec 27 12:22:11.525785756 (524h53m11.396868801s) [noise=true , address=45.91.116.14:4192 , piece_id=NXXJWPGS3Y72KS53TIRJI444Q74O6USMQMQ6MMF4HYBOTYUOYVCA, seg_pos={0 0}, upload=460428]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 27 12:22:11.525937154 (524h53m11.397020239s) [noise=true , address=45.91.116.14:4192 , piece_id=NXXJWPGS3Y72KS53TIRJI444Q74O6USMQMQ6MMF4HYBOTYUOYVCA, seg_pos={0 0}, upload=460428]: Long tail cancellation due to optimal threshold Dec 27 12:22:11.526409894 (524h53m11.397492949s) [noise=true , address=77.90.31.11:4553 , piece_id=IEDCEK4LCDFX7ODJL2RIPT2F7ZOK3R4DCRUKPH7C6FULCGHCWTLA, seg_pos={0 0}, upload=460428]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 27 12:22:11.526419713 (524h53m11.397502748s) [noise=true , address=77.90.31.11:4553 , piece_id=IEDCEK4LCDFX7ODJL2RIPT2F7ZOK3R4DCRUKPH7C6FULCGHCWTLA, seg_pos={0 0}, upload=460428]: Long tail cancellation due to optimal threshold Dec 27 12:22:11.526582983 (524h53m11.397666028s) [noise=true , address=77.90.31.21:4880 , piece_id=URUDPQFHT7SZBWVLRJXTL7EITRJF2C7BMNX6K5U4WBCIOVRUSDBQ, seg_pos={0 0}, upload=460428]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: write tcp 188.166.225.170:49966->77.90.31.21:4880: use of closed network connection; write tcp 188.166.225.170:49966->77.90.31.21:4880: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49966->77.90.31.21:4880: use of closed network connection Dec 27 12:22:11.526591871 (524h53m11.397674916s) [noise=true , address=77.90.31.21:4880 , piece_id=URUDPQFHT7SZBWVLRJXTL7EITRJF2C7BMNX6K5U4WBCIOVRUSDBQ, seg_pos={0 0}, upload=460428]: Long tail cancellation due to optimal threshold Dec 27 12:22:11.526614463 (524h53m11.397697508s) [seg_pos={0 0}, upload=460428]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 12:22:11.526704736 (524h53m11.397787781s) [upload=460428]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 12:22:11.526711729 (524h53m11.397794774s) [upload=460428]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 12:22:11.823700059 (524h53m11.694783094s) [upload=460428]: Done uploading segments... Dec 27 12:22:11.823727791 (524h53m11.694810836s) [upload=460428]: Upload finished. err: <nil> Duration: 6.075564098s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................X.................X..........] duration: 42.86547139s trace: https://jaeger.dev.storj.io/trace/3b6d882384466243 FAIL (2025-12-27 12:22:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 13QCH91kRMQjVj2ASsVDcyXQ1aapQRVeUm6MMCGimsR6Ns973h, Piece ID: EP2667PIQIHUWPTWD4REAWTQFSC3VR7V7VZQ344XJH4ASSJP4EMA) context deadline exceeded; piecestore: (Node ID: 126Sou5pnMm2eE2FJpv6WYX2VSqJvZZf9rFFJD6hFYDetRD28zu, Piece ID: Y2CDTLJ5S6DCNVTSXNOYT6FU7TSJYREG2BTIDPMOZAQ6FDXHQYAA) context deadline exceeded; piecestore: (Node ID: 12eLKUg2vTnZMrLaraSPEoGXe3jDVBJYyURdcn4f2gCN5pvvCQ8, Piece ID: 3DFZTINQGUUTX3YDEXERPDRZNN5CKAWOMVAQHL23ES2U5YHH2RAA) context deadline exceeded; piecestore: (Node ID: 1X5A7bLqMEQuaMbf9xZ7PDUD4161sz2QqdM9r59y3bsyPBJhaK, Piece ID: 65WTA5P6FPBVAHUJEAX3TIFC23M3A52IZ6FLARKO5RPSSIBZEXWQ) context deadline exceeded ======LOGS====== 38:4484 , piece_id=4US2ZAFTL4NOMWPTAV6ABOM4GE673PJHVJ7C5B3YGYDVFQUPQLXA, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.326171965 (524h44m33.197255s) [noise=true , address=77.90.31.12:4603 , piece_id=Y2CDTLJ5S6DCNVTSXNOYT6FU7TSJYREG2BTIDPMOZAQ6FDXHQYAA, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.326204638 (524h44m33.197287683s) [noise=true , address=77.90.31.13:4644 , piece_id=EP2667PIQIHUWPTWD4REAWTQFSC3VR7V7VZQ344XJH4ASSJP4EMA, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.326588799 (524h44m33.197671844s) [noise=true , address=145.79.227.44:4213 , piece_id=LE27IRRSIIE7T7JQLEMMW4DNJ3XMSRFOPE2DF2XV5XCNWZU3HDEQ, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.326927453 (524h44m33.198010498s) [noise=true , address=77.90.31.15:4721 , piece_id=3A72PLJ5CH6JGOMDL4PUYK3DVIUXRDHVQQA3WA2RW77FXKY46YOQ, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.326992878 (524h44m33.198075923s) [noise=true , address=145.79.227.32:4860 , piece_id=DXII7EZHDNAX6XTRBF6T2N2LTNP5H3DPHQEGDWKN7ZL7V3MVLDXQ, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.327699914 (524h44m33.198782949s) [noise=true , address=45.91.116.12:4258 , piece_id=3DFZTINQGUUTX3YDEXERPDRZNN5CKAWOMVAQHL23ES2U5YHH2RAA, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.327720574 (524h44m33.198803609s) [noise=true , address=145.79.227.34:4914 , piece_id=J7JR3XUPHJI3K4TZ23NECWAGL6YLFURO6MB7QR64L7LCEIBMWUXA, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.328318352 (524h44m33.199401387s) [noise=true , address=77.90.31.16:4746 , piece_id=UJJK5KRSEB3JCBXVUSE4TDHJ5JMIEDIGT4EWETKJCBEE46HSM2VA, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.328412561 (524h44m33.199495616s) [noise=true , address=145.79.227.30:4775 , piece_id=D7SJXV4WU5ZGJVYZHZ2UOVAF72ANYWSOCS5BH5U3Z3EI7MXW536A, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.328792936 (524h44m33.199875981s) [noise=true , address=77.90.31.11:4558 , piece_id=HGCZHZGYN6OPOFJJEIMQG2LXMGA6JDHIDDN3RTRF7XSZ56OL3M7Q, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.329056156 (524h44m33.200139191s) [noise=true , address=77.90.31.21:4905 , piece_id=65WTA5P6FPBVAHUJEAX3TIFC23M3A52IZ6FLARKO5RPSSIBZEXWQ, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.329239847 (524h44m33.200322892s) [noise=true , address=64.49.13.169:4709 , piece_id=XOM2255ZNKHB3I26XEYPYVMPCX4QTYGKL6NLXQOPOZ4PQCELVEAQ, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.329737634 (524h44m33.20082068s) [noise=true , address=145.79.227.40:4072 , piece_id=56RK7LN3GVLMGIYZD6FQ2FU23ADTGNWUYMJUMIHH5HHKATLYOQWQ, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.330431346 (524h44m33.201514391s) [noise=true , address=45.91.116.9:4287 , piece_id=F3TDAQGQWDLWZL2WSKWKMMB77TCNBHLIZVY6NPM5MX2ARUSVW37Q, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:34.611654557 (524h44m34.482737602s) [upload=460306]: Next returned nil segment Dec 27 12:13:34.611669636 (524h44m34.482752681s) [upload=460306]: Waiting for error group managing segments... Dec 27 12:13:34.940178727 (524h44m34.811261762s) [noise=true , address=64.49.13.138:4484 , piece_id=4US2ZAFTL4NOMWPTAV6ABOM4GE673PJHVJ7C5B3YGYDVFQUPQLXA, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.218550166 (524h44m35.089633211s) [noise=true , address=145.79.227.32:4860 , piece_id=DXII7EZHDNAX6XTRBF6T2N2LTNP5H3DPHQEGDWKN7ZL7V3MVLDXQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.226012146 (524h44m35.097095191s) [noise=true , address=145.79.227.34:4914 , piece_id=J7JR3XUPHJI3K4TZ23NECWAGL6YLFURO6MB7QR64L7LCEIBMWUXA, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.400566879 (524h44m35.271649914s) [noise=true , address=145.79.227.33:4891 , piece_id=2R47TJ3SLIMPLH4JZTSINHBZXQ3JJKXVETIMILFATRXEI6DZLJLQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.404347268 (524h44m35.275430303s) [noise=true , address=77.90.31.21:4905 , piece_id=65WTA5P6FPBVAHUJEAX3TIFC23M3A52IZ6FLARKO5RPSSIBZEXWQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.416074639 (524h44m35.287157674s) [noise=true , address=77.90.31.13:4644 , piece_id=EP2667PIQIHUWPTWD4REAWTQFSC3VR7V7VZQ344XJH4ASSJP4EMA, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.452194618 (524h44m35.323277663s) [noise=true , address=45.91.116.9:4287 , piece_id=F3TDAQGQWDLWZL2WSKWKMMB77TCNBHLIZVY6NPM5MX2ARUSVW37Q, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.525559618 (524h44m35.396642663s) [noise=true , address=45.91.116.12:4258 , piece_id=3DFZTINQGUUTX3YDEXERPDRZNN5CKAWOMVAQHL23ES2U5YHH2RAA, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.710902755 (524h44m35.58198579s) [noise=true , address=77.90.31.12:4603 , piece_id=Y2CDTLJ5S6DCNVTSXNOYT6FU7TSJYREG2BTIDPMOZAQ6FDXHQYAA, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.833345128 (524h44m35.704428173s) [noise=true , address=77.90.31.15:4721 , piece_id=3A72PLJ5CH6JGOMDL4PUYK3DVIUXRDHVQQA3WA2RW77FXKY46YOQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:36.102928202 (524h44m35.974011237s) [noise=true , address=145.79.227.44:4213 , piece_id=LE27IRRSIIE7T7JQLEMMW4DNJ3XMSRFOPE2DF2XV5XCNWZU3HDEQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:36.153147128 (524h44m36.024230183s) [noise=true , address=145.79.227.40:4072 , piece_id=56RK7LN3GVLMGIYZD6FQ2FU23ADTGNWUYMJUMIHH5HHKATLYOQWQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:36.208908849 (524h44m36.079991884s) [noise=true , address=145.79.227.30:4775 , piece_id=D7SJXV4WU5ZGJVYZHZ2UOVAF72ANYWSOCS5BH5U3Z3EI7MXW536A, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:36.410529180 (524h44m36.281612215s) [noise=true , address=45.91.116.12:4248 , piece_id=G5SOGWARFYFBXQRSRB5OXZZEKN6BMGN3PQC2RBB7JHN55QUPRXIQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:36.410547656 (524h44m36.281630691s) [seg_pos={0 0}, upload=460306]: Segment reached optimal threshold of 14 pieces. Dec 27 12:13:36.410724162 (524h44m36.281807197s) [noise=true , address=77.90.31.11:4558 , piece_id=HGCZHZGYN6OPOFJJEIMQG2LXMGA6JDHIDDN3RTRF7XSZ56OL3M7Q, seg_pos={0 0}, upload=460306]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mGEP1n1icC2gEkyRVy6BeM4vsrgqYWaQeB5rUtxjrWQjyFhTS): protocol: expected piece hash; context canceled; EOF Dec 27 12:13:36.410730103 (524h44m36.281813138s) [noise=true , address=77.90.31.11:4558 , piece_id=HGCZHZGYN6OPOFJJEIMQG2LXMGA6JDHIDDN3RTRF7XSZ56OL3M7Q, seg_pos={0 0}, upload=460306]: Long tail cancellation due to optimal threshold Dec 27 12:13:36.410916448 (524h44m36.281999483s) [noise=true , address=77.90.31.16:4746 , piece_id=UJJK5KRSEB3JCBXVUSE4TDHJ5JMIEDIGT4EWETKJCBEE46HSM2VA, seg_pos={0 0}, upload=460306]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o): protocol: write tcp 188.166.225.170:42390->77.90.31.16:4746: use of closed network connection; write tcp 188.166.225.170:42390->77.90.31.16:4746: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42390->77.90.31.16:4746: use of closed network connection Dec 27 12:13:36.410922750 (524h44m36.282005785s) [noise=true , address=77.90.31.16:4746 , piece_id=UJJK5KRSEB3JCBXVUSE4TDHJ5JMIEDIGT4EWETKJCBEE46HSM2VA, seg_pos={0 0}, upload=460306]: Long tail cancellation due to optimal threshold Dec 27 12:13:36.411020866 (524h44m36.282103911s) [noise=true , address=193.149.164.143:4699 , piece_id=4A5T3OSNSHNKNXKRWOWVJC222TNAISNRGN3NCXQGVFKNHVDBKXHQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LncNSBhhreg9JTByJPSB7eGUEu88Eyd3jc9pg24VsEjp4FULq): protocol: expected piece hash; context canceled; EOF Dec 27 12:13:36.411036306 (524h44m36.282119341s) [noise=true , address=193.149.164.143:4699 , piece_id=4A5T3OSNSHNKNXKRWOWVJC222TNAISNRGN3NCXQGVFKNHVDBKXHQ, seg_pos={0 0}, upload=460306]: Long tail cancellation due to optimal threshold Dec 27 12:13:36.411086592 (524h44m36.282169637s) [noise=true , address=64.49.13.169:4709 , piece_id=XOM2255ZNKHB3I26XEYPYVMPCX4QTYGKL6NLXQOPOZ4PQCELVEAQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:58550->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:58550->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58550->64.49.13.169:4709: use of closed network connection Dec 27 12:13:36.411093095 (524h44m36.28217613s) [noise=true , address=64.49.13.169:4709 , piece_id=XOM2255ZNKHB3I26XEYPYVMPCX4QTYGKL6NLXQOPOZ4PQCELVEAQ, seg_pos={0 0}, upload=460306]: Long tail cancellation due to optimal threshold Dec 27 12:13:36.411118663 (524h44m36.282201708s) [seg_pos={0 0}, upload=460306]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 12:13:36.411202012 (524h44m36.282285047s) [upload=460306]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 12:13:36.411207102 (524h44m36.282290137s) [upload=460306]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 12:13:36.701956341 (524h44m36.573039386s) [upload=460306]: Done uploading segments... Dec 27 12:13:36.701976790 (524h44m36.573059825s) [upload=460306]: Upload finished. err: <nil> Duration: 42.865494684s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [X....XX.X....XX......X.X.X.XX.X....X.X..X..XXXXXXX.X.XX...XX] duration: 30.003949056s trace: https://jaeger.dev.storj.io/trace/3085452edd720d63 FAIL (2025-12-27 12:22:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: TNYFLGHLEPYUXKB4UJ4GL2SSQSEZFEL3IZ3NVB6A2XYRZDAAVBWQ) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: 3BGF2ARRRGA4WZGFVBQQGVEMYUGC66TGQ65Q5SIONKINMNNOCVIA) context deadline exceeded ======LOGS====== Dec 27 12:22:09.025089235 (524h53m8.89617227s) [upload=460428]: Starting upload Dec 27 12:22:09.025170170 (524h53m8.896253215s) [upload=460428]: Done starting upload Dec 27 12:22:09.025359219 (524h53m8.896442265s) [upload=460428]: Uploading segments... Dec 27 12:22:09.025394326 (524h53m8.896477371s) [upload=460428]: Got next segment. Inline: false Dec 27 12:22:09.025418753 (524h53m8.896501798s) [upload=460428]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 12:22:09.025423201 (524h53m8.896506237s) [upload=460428]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 12:22:09.268893583 (524h53m9.139976628s) [seg_pos={0 0}, upload=460428]: Begin upload segment... Dec 27 12:22:09.268944200 (524h53m9.140027235s) [seg_pos={0 0}, upload=460428]: Done begin upload segment. Dec 27 12:22:09.269000027 (524h53m9.140083062s) [noise=true , address=145.79.227.35:4481 , piece_id=BDOVCOR7TNCF4XKG2DKNOCHCKQ6535MR3B7GDFMTVZDAITPUK5GA, seg_pos={0 0}, upload=460428]: Uploading piece... Dec 27 12:22:09.269027208 (524h53m9.140110243s) [noise=true , address=145.79.227.36:4519 , piece_id=XSNAEIAUAGSJLUQ7E2N3CEW3Z6NRKYEXM3IOBJWDYPUM4IKWFIQQ, seg_pos={0 0}, upload=460428]: Uploading piece... Dec 27 12:22:09.269609176 (524h53m9.140692211s) [noise=true , address=45.91.116.10:4084 , piece_id=3BGF2ARRRGA4WZGFVBQQGVEMYUGC66TGQ65Q5SIONKINMNNOCVIA, seg_pos={0 0}, upload=460428]: Uploading piece... Dec 27 12:22:09.269984872 (524h53m9.141067907s) [noise=true , address=77.90.31.11:4553 , piece_id=IEDCEK4LCDFX7ODJL2RIPT2F7ZOK3R4DCRUKPH7C6FULCGHCWTLA, seg_pos={0 0}, upload=460428]: Uploading piece... Dec 27 12:22:09.270586687 (524h53m9.141669722s) [noise=true , address=45.91.116.14:4192 , piece_id=NXXJWPGS3Y72KS53TIRJI444Q74O6USMQMQ6MMF4HYBOTYUOYVCA, seg_pos={0 0}, upload=460428]: Uploading piece... Dec 27 12:22:09.271050620 (524h53m9.142133665s) [noise=true , address=77.90.31.21:4880 , piece_id=URUDPQFHT7SZBWVLRJXTL7EITRJF2C7BMNX6K5U4WBCIOVRUSDBQ, seg_pos={0 0}, upload=460428]: Uploading piece... Dec 27 12:22:09.271699396 (524h53m9.142782431s) [noise=true , address=145.79.227.4:4185 , piece_id=EUE6RNGIPQ2HAX6CBCI55ANFIA3ITZB4T4PTZ7SNM2KEMQWN47IQ, seg_pos={0 0}, upload=460428]: Uploading piece... Dec 27 12:22:09.273021724 (524h53m9.144104759s) [noise=true , address=145.79.227.45:4229 , piece_id=CNTCTWZPN7CWXUUPSQ2S2I2POUUS3AJWH47ZJ6LRE3HI7QATWGMA, seg_pos={0 0}, upload=460428]: Uploading piece... Dec 27 12:22:09.273296688 (524h53m9.144379723s) [noise=true , address=145.79.227.14:4837 , piece_id=E5G54MM2XSSNWXJPMWJIEZAKXB6ZSFU6IVUBZQOJH6SAGYPJDWLA, seg_pos={0 0}, upload=460428]: Uploading piece... Dec 27 12:22:09.274364952 (524h53m9.145447997s) [noise=true , address=145.79.227.10:4402 , piece_id=YTRTL32F5NBVGJLK47LR2VF3V7MP6LNOCFIIMEAV27TZ2WJYDBKA, seg_pos={0 0}, upload=460428]: Uploading piece... Dec 27 12:22:09.274771416 (524h53m9.145854451s) [noise=true , address=145.79.227.42:4123 , piece_id=3IZULC56BEE75VT66DSM5HCQN5R6H2LS2Y627VJHPBCOO5WWUTOA, seg_pos={0 0}, upload=460428]: Uploading piece... Dec 27 12:22:09.276445052 (524h53m9.147528097s) [noise=true , address=145.79.227.25:4617 , piece_id=4S2ISYTU55S4NEGPOEBYAVAIT5GBD47HQ57226QPTGGHJODVTCYQ, seg_pos={0 0}, upload=460428]: Uploading piece... Dec 27 12:22:09.278172502 (524h53m9.149255547s) [noise=true , address=45.91.116.12:4227 , piece_id=TNYFLGHLEPYUXKB4UJ4GL2SSQSEZFEL3IZ3NVB6A2XYRZDAAVBWQ, seg_pos={0 0}, upload=460428]: Uploading piece... Dec 27 12:22:09.278988586 (524h53m9.150071631s) [noise=true , address=145.79.227.5:4221 , piece_id=FOV2F75NL4HXMPXGKSV5YKTYAANYUMEO7HDM2GLAZ5AT6WD6M6SA, seg_pos={0 0}, upload=460428]: Uploading piece... Dec 27 12:22:09.287289603 (524h53m9.158372638s) [noise=true , address=145.79.227.37:4567 , piece_id=PT3AQZ47CLHEHB3SOQFISBAJNGCGPNR2ZVA7GO53KAQH6A4H5QFA, seg_pos={0 0}, upload=460428]: Uploading piece... Dec 27 12:22:09.292163203 (524h53m9.163246258s) [noise=true , address=145.79.227.2:4113 , piece_id=RDT6KTALFOHFEKCLIW2YTUF3ZTM3KBYTVHXVII3PW7QVJOSLZRWA, seg_pos={0 0}, upload=460428]: Uploading piece... Dec 27 12:22:11.001147820 (524h53m10.872230875s) [upload=460428]: Next returned nil segment Dec 27 12:22:11.001168629 (524h53m10.872251674s) [upload=460428]: Waiting for error group managing segments... Dec 27 12:22:11.290317381 (524h53m11.161400446s) [noise=true , address=145.79.227.45:4229 , piece_id=CNTCTWZPN7CWXUUPSQ2S2I2POUUS3AJWH47ZJ6LRE3HI7QATWGMA, seg_pos={0 0}, upload=460428]: Done uploading piece. err: <nil> Dec 27 12:22:11.294623030 (524h53m11.165706085s) [noise=true , address=145.79.227.35:4481 , piece_id=BDOVCOR7TNCF4XKG2DKNOCHCKQ6535MR3B7GDFMTVZDAITPUK5GA, seg_pos={0 0}, upload=460428]: Done uploading piece. err: <nil> Dec 27 12:22:11.294785650 (524h53m11.165868695s) [noise=true , address=145.79.227.42:4123 , piece_id=3IZULC56BEE75VT66DSM5HCQN5R6H2LS2Y627VJHPBCOO5WWUTOA, seg_pos={0 0}, upload=460428]: Done uploading piece. err: <nil> Dec 27 12:22:11.295115197 (524h53m11.166198232s) [noise=true , address=145.79.227.10:4402 , piece_id=YTRTL32F5NBVGJLK47LR2VF3V7MP6LNOCFIIMEAV27TZ2WJYDBKA, seg_pos={0 0}, upload=460428]: Done uploading piece. err: <nil> Dec 27 12:22:11.295306602 (524h53m11.166389647s) [noise=true , address=145.79.227.25:4617 , piece_id=4S2ISYTU55S4NEGPOEBYAVAIT5GBD47HQ57226QPTGGHJODVTCYQ, seg_pos={0 0}, upload=460428]: Done uploading piece. err: <nil> Dec 27 12:22:11.295899040 (524h53m11.166982085s) [noise=true , address=145.79.227.2:4113 , piece_id=RDT6KTALFOHFEKCLIW2YTUF3ZTM3KBYTVHXVII3PW7QVJOSLZRWA, seg_pos={0 0}, upload=460428]: Done uploading piece. err: <nil> Dec 27 12:22:11.311065399 (524h53m11.182148444s) [noise=true , address=145.79.227.36:4519 , piece_id=XSNAEIAUAGSJLUQ7E2N3CEW3Z6NRKYEXM3IOBJWDYPUM4IKWFIQQ, seg_pos={0 0}, upload=460428]: Done uploading piece. err: <nil> Dec 27 12:22:11.311517950 (524h53m11.182600985s) [noise=true , address=145.79.227.4:4185 , piece_id=EUE6RNGIPQ2HAX6CBCI55ANFIA3ITZB4T4PTZ7SNM2KEMQWN47IQ, seg_pos={0 0}, upload=460428]: Done uploading piece. err: <nil> Dec 27 12:22:11.313099943 (524h53m11.184182988s) [noise=true , address=145.79.227.5:4221 , piece_id=FOV2F75NL4HXMPXGKSV5YKTYAANYUMEO7HDM2GLAZ5AT6WD6M6SA, seg_pos={0 0}, upload=460428]: Done uploading piece. err: <nil> Dec 27 12:22:11.318998825 (524h53m11.19008186s) [noise=true , address=145.79.227.14:4837 , piece_id=E5G54MM2XSSNWXJPMWJIEZAKXB6ZSFU6IVUBZQOJH6SAGYPJDWLA, seg_pos={0 0}, upload=460428]: Done uploading piece. err: <nil> Dec 27 12:22:11.323220775 (524h53m11.194303851s) [noise=true , address=145.79.227.37:4567 , piece_id=PT3AQZ47CLHEHB3SOQFISBAJNGCGPNR2ZVA7GO53KAQH6A4H5QFA, seg_pos={0 0}, upload=460428]: Done uploading piece. err: <nil> Dec 27 12:22:11.498870103 (524h53m11.369953148s) [noise=true , address=45.91.116.10:4084 , piece_id=3BGF2ARRRGA4WZGFVBQQGVEMYUGC66TGQ65Q5SIONKINMNNOCVIA, seg_pos={0 0}, upload=460428]: Done uploading piece. err: <nil> Dec 27 12:22:11.525496124 (524h53m11.396579159s) [noise=true , address=45.91.116.12:4227 , piece_id=TNYFLGHLEPYUXKB4UJ4GL2SSQSEZFEL3IZ3NVB6A2XYRZDAAVBWQ, seg_pos={0 0}, upload=460428]: Done uploading piece. err: <nil> Dec 27 12:22:11.525519288 (524h53m11.396602333s) [seg_pos={0 0}, upload=460428]: Segment reached optimal threshold of 13 pieces. Dec 27 12:22:11.525785756 (524h53m11.396868801s) [noise=true , address=45.91.116.14:4192 , piece_id=NXXJWPGS3Y72KS53TIRJI444Q74O6USMQMQ6MMF4HYBOTYUOYVCA, seg_pos={0 0}, upload=460428]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 27 12:22:11.525937154 (524h53m11.397020239s) [noise=true , address=45.91.116.14:4192 , piece_id=NXXJWPGS3Y72KS53TIRJI444Q74O6USMQMQ6MMF4HYBOTYUOYVCA, seg_pos={0 0}, upload=460428]: Long tail cancellation due to optimal threshold Dec 27 12:22:11.526409894 (524h53m11.397492949s) [noise=true , address=77.90.31.11:4553 , piece_id=IEDCEK4LCDFX7ODJL2RIPT2F7ZOK3R4DCRUKPH7C6FULCGHCWTLA, seg_pos={0 0}, upload=460428]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 27 12:22:11.526419713 (524h53m11.397502748s) [noise=true , address=77.90.31.11:4553 , piece_id=IEDCEK4LCDFX7ODJL2RIPT2F7ZOK3R4DCRUKPH7C6FULCGHCWTLA, seg_pos={0 0}, upload=460428]: Long tail cancellation due to optimal threshold Dec 27 12:22:11.526582983 (524h53m11.397666028s) [noise=true , address=77.90.31.21:4880 , piece_id=URUDPQFHT7SZBWVLRJXTL7EITRJF2C7BMNX6K5U4WBCIOVRUSDBQ, seg_pos={0 0}, upload=460428]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: write tcp 188.166.225.170:49966->77.90.31.21:4880: use of closed network connection; write tcp 188.166.225.170:49966->77.90.31.21:4880: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49966->77.90.31.21:4880: use of closed network connection Dec 27 12:22:11.526591871 (524h53m11.397674916s) [noise=true , address=77.90.31.21:4880 , piece_id=URUDPQFHT7SZBWVLRJXTL7EITRJF2C7BMNX6K5U4WBCIOVRUSDBQ, seg_pos={0 0}, upload=460428]: Long tail cancellation due to optimal threshold Dec 27 12:22:11.526614463 (524h53m11.397697508s) [seg_pos={0 0}, upload=460428]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 12:22:11.526704736 (524h53m11.397787781s) [upload=460428]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 12:22:11.526711729 (524h53m11.397794774s) [upload=460428]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 12:22:11.823700059 (524h53m11.694783094s) [upload=460428]: Done uploading segments... Dec 27 12:22:11.823727791 (524h53m11.694810836s) [upload=460428]: Upload finished. err: <nil> Duration: 30.003960568s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 2.298277487s trace: https://jaeger.dev.storj.io/trace/6dd289079d7aa430 FAIL (2025-12-27 12:22:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.298306794s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 6.85627925s trace: https://jaeger.dev.storj.io/trace/578b4df2b1bacee FAIL (2025-12-27 12:22:30 UTC) orbiter warning: context deadline exceeded Duration: 6.856315769s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...X........................................................] duration: 4.70592754s trace: https://jaeger.dev.storj.io/trace/12569321705fed16 FAIL (2025-12-27 12:22:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.7059538s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [.X....XX.X....XX......X.X.X.XX.X....X.X..X..XXXXXXX.X.XX...X] duration: 30.003605241s trace: https://jaeger.dev.storj.io/trace/7af2b13f9bcd6869 FAIL (2025-12-27 12:22:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: ABY5UKOEJ3OPXBHQ5H6Q7Q6V2HOGW2GFFHCD3Y2P72ENMJ62653Q) context deadline exceeded; piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: RB27NTVZN6MXJL7KFRFGIFLQ2VY7GT3G7XHU5OCLQIKOKVU3IPWA) context deadline exceeded; piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: X2PTBVE4OMEOOMONID7JH3WDO2LRSLCSJEVCRDHQXJOLWY465EQQ) context deadline exceeded ======LOGS====== 12:21:38.769311000 (524h52m38.640394045s) [upload=460416]: Starting upload Dec 27 12:21:38.769402916 (524h52m38.640485961s) [upload=460416]: Done starting upload Dec 27 12:21:38.769618606 (524h52m38.640701651s) [upload=460416]: Uploading segments... Dec 27 12:21:38.769741982 (524h52m38.640825027s) [upload=460416]: Got next segment. Inline: false Dec 27 12:21:38.769888330 (524h52m38.640971365s) [upload=460416]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 12:21:38.769896335 (524h52m38.64097937s) [upload=460416]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 12:21:39.017103003 (524h52m38.888186048s) [seg_pos={0 0}, upload=460416]: Begin upload segment... Dec 27 12:21:39.017155673 (524h52m38.888238718s) [seg_pos={0 0}, upload=460416]: Done begin upload segment. Dec 27 12:21:39.017193665 (524h52m38.88827672s) [noise=true , address=145.79.227.4:4186 , piece_id=5VA4XHOMB3GW4UZPG5KJOAI6B57RPWVH5XCKNLXHKRPY3NWRX2WA, seg_pos={0 0}, upload=460416]: Uploading piece... Dec 27 12:21:39.017242579 (524h52m38.888325614s) [noise=true , address=45.91.116.4:4939 , piece_id=VX3PSUE5UXVYUJMR7MXBQBKEA26IC2TRH75GS2QCEHMPTSONM5YQ, seg_pos={0 0}, upload=460416]: Uploading piece... Dec 27 12:21:39.017973100 (524h52m38.889056145s) [noise=true , address=77.90.31.10:4518 , piece_id=ABY5UKOEJ3OPXBHQ5H6Q7Q6V2HOGW2GFFHCD3Y2P72ENMJ62653Q, seg_pos={0 0}, upload=460416]: Uploading piece... Dec 27 12:21:39.018254305 (524h52m38.88933734s) [noise=true , address=45.91.116.12:4228 , piece_id=X2PTBVE4OMEOOMONID7JH3WDO2LRSLCSJEVCRDHQXJOLWY465EQQ, seg_pos={0 0}, upload=460416]: Uploading piece... Dec 27 12:21:39.018502077 (524h52m38.889585112s) [noise=true , address=145.79.227.12:4765 , piece_id=3FKW2K3BTSC2H62BK43RKHROWOP5MIUETRGZ3WITYXZXYDDZQOFA, seg_pos={0 0}, upload=460416]: Uploading piece... Dec 27 12:21:39.018682330 (524h52m38.889765365s) [noise=true , address=77.90.31.19:4841 , piece_id=SEWE67XZVNFH5A4OXV6SNRLMBZOW2PORYL5KPWX5LGA6SUXSZWHA, seg_pos={0 0}, upload=460416]: Uploading piece... Dec 27 12:21:39.018913480 (524h52m38.889996525s) [noise=true , address=145.79.227.36:4517 , piece_id=34TS6WKSDJGXLNOZN6XPJ4UGP4OOIFS42B5KRDW3TBMIBVDA75LQ, seg_pos={0 0}, upload=460416]: Uploading piece... Dec 27 12:21:39.019365952 (524h52m38.890448997s) [noise=true , address=45.91.116.13:4154 , piece_id=RB27NTVZN6MXJL7KFRFGIFLQ2VY7GT3G7XHU5OCLQIKOKVU3IPWA, seg_pos={0 0}, upload=460416]: Uploading piece... Dec 27 12:21:39.020080502 (524h52m38.891163537s) [noise=true , address=145.79.227.46:4602 , piece_id=IUDAMGEMLVQTQERFJ2XBO5C3VDN5HRCD5MSZW6SEMKJA6DZOC6EQ, seg_pos={0 0}, upload=460416]: Uploading piece... Dec 27 12:21:39.020325238 (524h52m38.891408273s) [noise=true , address=145.79.227.17:4971 , piece_id=2UCPXWU2ACTYQGKNNI27XQ4IHIH7GMXAKIHZY64IHBNQ6BT3YIZQ, seg_pos={0 0}, upload=460416]: Uploading piece... Dec 27 12:21:39.020706424 (524h52m38.891789459s) [noise=true , address=45.91.116.14:4191 , piece_id=SHZEZMLUDRXFIGKCFV7XKL7FQTI2TBEU3PDS6MKP2FGJGGMBBU3A, seg_pos={0 0}, upload=460416]: Uploading piece... Dec 27 12:21:39.021190215 (524h52m38.89227325s) [noise=true , address=145.79.227.2:4114 , piece_id=AVV7NBYDS2BNM3EPVBOIIE3KEYVXM5VQHOEUTFWE3HFUWXKFMZDA, seg_pos={0 0}, upload=460416]: Uploading piece... Dec 27 12:21:39.021702932 (524h52m38.892785967s) [noise=true , address=145.79.227.23:4546 , piece_id=5DXZMFJQJN4TZBI625HPMYC634EHVYO2WXUWFDNTYOZ76K2MN5PA, seg_pos={0 0}, upload=460416]: Uploading piece... Dec 27 12:21:39.022247379 (524h52m38.893330414s) [noise=true , address=145.79.227.39:4014 , piece_id=BG3ENDADSK3J237BBHWRE3SOB6QYRFS2IIGGF36POLJGH4ZTMD2Q, seg_pos={0 0}, upload=460416]: Uploading piece... Dec 27 12:21:39.022402343 (524h52m38.893485379s) [noise=true , address=45.91.116.6:4975 , piece_id=IG5FXL5HHGYYT2PQ33I6EUE5M4JIAALSWQUPQK7XUABLBVFKKBVQ, seg_pos={0 0}, upload=460416]: Uploading piece... Dec 27 12:21:39.022828104 (524h52m38.893911139s) [noise=true , address=45.91.116.8:4046 , piece_id=6HROBBMDHKX45YLJTY3GNCG3VXMTLAJVJJ5F36FWGN4AWRQLX3NA, seg_pos={0 0}, upload=460416]: Uploading piece... Dec 27 12:21:40.734034321 (524h52m40.605117366s) [upload=460416]: Next returned nil segment Dec 27 12:21:40.734166893 (524h52m40.605249928s) [upload=460416]: Waiting for error group managing segments... Dec 27 12:21:41.019893920 (524h52m40.890976965s) [noise=true , address=145.79.227.2:4114 , piece_id=AVV7NBYDS2BNM3EPVBOIIE3KEYVXM5VQHOEUTFWE3HFUWXKFMZDA, seg_pos={0 0}, upload=460416]: Done uploading piece. err: <nil> Dec 27 12:21:41.024075622 (524h52m40.895158667s) [noise=true , address=145.79.227.12:4765 , piece_id=3FKW2K3BTSC2H62BK43RKHROWOP5MIUETRGZ3WITYXZXYDDZQOFA, seg_pos={0 0}, upload=460416]: Done uploading piece. err: <nil> Dec 27 12:21:41.024646890 (524h52m40.895729925s) [noise=true , address=145.79.227.46:4602 , piece_id=IUDAMGEMLVQTQERFJ2XBO5C3VDN5HRCD5MSZW6SEMKJA6DZOC6EQ, seg_pos={0 0}, upload=460416]: Done uploading piece. err: <nil> Dec 27 12:21:41.025335130 (524h52m40.896418166s) [noise=true , address=145.79.227.17:4971 , piece_id=2UCPXWU2ACTYQGKNNI27XQ4IHIH7GMXAKIHZY64IHBNQ6BT3YIZQ, seg_pos={0 0}, upload=460416]: Done uploading piece. err: <nil> Dec 27 12:21:41.029527693 (524h52m40.900610728s) [noise=true , address=145.79.227.39:4014 , piece_id=BG3ENDADSK3J237BBHWRE3SOB6QYRFS2IIGGF36POLJGH4ZTMD2Q, seg_pos={0 0}, upload=460416]: Done uploading piece. err: <nil> Dec 27 12:21:41.032834240 (524h52m40.903917275s) [noise=true , address=145.79.227.23:4546 , piece_id=5DXZMFJQJN4TZBI625HPMYC634EHVYO2WXUWFDNTYOZ76K2MN5PA, seg_pos={0 0}, upload=460416]: Done uploading piece. err: <nil> Dec 27 12:21:41.040904537 (524h52m40.911987572s) [noise=true , address=145.79.227.36:4517 , piece_id=34TS6WKSDJGXLNOZN6XPJ4UGP4OOIFS42B5KRDW3TBMIBVDA75LQ, seg_pos={0 0}, upload=460416]: Done uploading piece. err: <nil> Dec 27 12:21:41.049884757 (524h52m40.920967792s) [noise=true , address=145.79.227.4:4186 , piece_id=5VA4XHOMB3GW4UZPG5KJOAI6B57RPWVH5XCKNLXHKRPY3NWRX2WA, seg_pos={0 0}, upload=460416]: Done uploading piece. err: <nil> Dec 27 12:21:41.226937177 (524h52m41.098020242s) [noise=true , address=77.90.31.10:4518 , piece_id=ABY5UKOEJ3OPXBHQ5H6Q7Q6V2HOGW2GFFHCD3Y2P72ENMJ62653Q, seg_pos={0 0}, upload=460416]: Done uploading piece. err: <nil> Dec 27 12:21:41.243768578 (524h52m41.114851613s) [noise=true , address=45.91.116.13:4154 , piece_id=RB27NTVZN6MXJL7KFRFGIFLQ2VY7GT3G7XHU5OCLQIKOKVU3IPWA, seg_pos={0 0}, upload=460416]: Done uploading piece. err: <nil> Dec 27 12:21:41.252431492 (524h52m41.123514537s) [noise=true , address=45.91.116.12:4228 , piece_id=X2PTBVE4OMEOOMONID7JH3WDO2LRSLCSJEVCRDHQXJOLWY465EQQ, seg_pos={0 0}, upload=460416]: Done uploading piece. err: <nil> Dec 27 12:21:41.278977581 (524h52m41.150060626s) [noise=true , address=45.91.116.4:4939 , piece_id=VX3PSUE5UXVYUJMR7MXBQBKEA26IC2TRH75GS2QCEHMPTSONM5YQ, seg_pos={0 0}, upload=460416]: Done uploading piece. err: <nil> Dec 27 12:21:41.292121430 (524h52m41.163204485s) [noise=true , address=77.90.31.19:4841 , piece_id=SEWE67XZVNFH5A4OXV6SNRLMBZOW2PORYL5KPWX5LGA6SUXSZWHA, seg_pos={0 0}, upload=460416]: Done uploading piece. err: <nil> Dec 27 12:21:41.292141629 (524h52m41.163224674s) [seg_pos={0 0}, upload=460416]: Segment reached optimal threshold of 13 pieces. Dec 27 12:21:41.292356097 (524h52m41.163439132s) [noise=true , address=45.91.116.8:4046 , piece_id=6HROBBMDHKX45YLJTY3GNCG3VXMTLAJVJJ5F36FWGN4AWRQLX3NA, seg_pos={0 0}, upload=460416]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 27 12:21:41.292363641 (524h52m41.163446686s) [noise=true , address=45.91.116.8:4046 , piece_id=6HROBBMDHKX45YLJTY3GNCG3VXMTLAJVJJ5F36FWGN4AWRQLX3NA, seg_pos={0 0}, upload=460416]: Long tail cancellation due to optimal threshold Dec 27 12:21:41.292536440 (524h52m41.163619505s) [noise=true , address=45.91.116.6:4975 , piece_id=IG5FXL5HHGYYT2PQ33I6EUE5M4JIAALSWQUPQK7XUABLBVFKKBVQ, seg_pos={0 0}, upload=460416]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 27 12:21:41.292561789 (524h52m41.163644844s) [noise=true , address=45.91.116.6:4975 , piece_id=IG5FXL5HHGYYT2PQ33I6EUE5M4JIAALSWQUPQK7XUABLBVFKKBVQ, seg_pos={0 0}, upload=460416]: Long tail cancellation due to optimal threshold Dec 27 12:21:41.292635128 (524h52m41.163718253s) [noise=true , address=45.91.116.14:4191 , piece_id=SHZEZMLUDRXFIGKCFV7XKL7FQTI2TBEU3PDS6MKP2FGJGGMBBU3A, seg_pos={0 0}, upload=460416]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:47508->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:47508->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47508->45.91.116.14:4191: use of closed network connection Dec 27 12:21:41.292648984 (524h52m41.163732019s) [noise=true , address=45.91.116.14:4191 , piece_id=SHZEZMLUDRXFIGKCFV7XKL7FQTI2TBEU3PDS6MKP2FGJGGMBBU3A, seg_pos={0 0}, upload=460416]: Long tail cancellation due to optimal threshold Dec 27 12:21:41.292675715 (524h52m41.16375875s) [seg_pos={0 0}, upload=460416]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 12:21:41.292796666 (524h52m41.163879711s) [upload=460416]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 12:21:41.292813277 (524h52m41.163896322s) [upload=460416]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 12:21:41.588392434 (524h52m41.459475469s) [upload=460416]: Done uploading segments... Dec 27 12:21:41.588410798 (524h52m41.459493843s) [upload=460416]: Upload finished. err: <nil> Duration: 30.003630459s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 2.701268335s trace: https://jaeger.dev.storj.io/trace/6dd289079d7aa430 FAIL (2025-12-27 12:22:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.701458387s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................X.................X.........] duration: 46.892703722s trace: https://jaeger.dev.storj.io/trace/3b6d882384466243 FAIL (2025-12-27 12:21:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 13QCH91kRMQjVj2ASsVDcyXQ1aapQRVeUm6MMCGimsR6Ns973h, Piece ID: EP2667PIQIHUWPTWD4REAWTQFSC3VR7V7VZQ344XJH4ASSJP4EMA) context deadline exceeded; piecestore: (Node ID: 126Sou5pnMm2eE2FJpv6WYX2VSqJvZZf9rFFJD6hFYDetRD28zu, Piece ID: Y2CDTLJ5S6DCNVTSXNOYT6FU7TSJYREG2BTIDPMOZAQ6FDXHQYAA) context deadline exceeded; piecestore: (Node ID: 12eLKUg2vTnZMrLaraSPEoGXe3jDVBJYyURdcn4f2gCN5pvvCQ8, Piece ID: 3DFZTINQGUUTX3YDEXERPDRZNN5CKAWOMVAQHL23ES2U5YHH2RAA) context deadline exceeded; piecestore: (Node ID: 1X5A7bLqMEQuaMbf9xZ7PDUD4161sz2QqdM9r59y3bsyPBJhaK, Piece ID: 65WTA5P6FPBVAHUJEAX3TIFC23M3A52IZ6FLARKO5RPSSIBZEXWQ) context deadline exceeded ======LOGS====== 38:4484 , piece_id=4US2ZAFTL4NOMWPTAV6ABOM4GE673PJHVJ7C5B3YGYDVFQUPQLXA, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.326171965 (524h44m33.197255s) [noise=true , address=77.90.31.12:4603 , piece_id=Y2CDTLJ5S6DCNVTSXNOYT6FU7TSJYREG2BTIDPMOZAQ6FDXHQYAA, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.326204638 (524h44m33.197287683s) [noise=true , address=77.90.31.13:4644 , piece_id=EP2667PIQIHUWPTWD4REAWTQFSC3VR7V7VZQ344XJH4ASSJP4EMA, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.326588799 (524h44m33.197671844s) [noise=true , address=145.79.227.44:4213 , piece_id=LE27IRRSIIE7T7JQLEMMW4DNJ3XMSRFOPE2DF2XV5XCNWZU3HDEQ, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.326927453 (524h44m33.198010498s) [noise=true , address=77.90.31.15:4721 , piece_id=3A72PLJ5CH6JGOMDL4PUYK3DVIUXRDHVQQA3WA2RW77FXKY46YOQ, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.326992878 (524h44m33.198075923s) [noise=true , address=145.79.227.32:4860 , piece_id=DXII7EZHDNAX6XTRBF6T2N2LTNP5H3DPHQEGDWKN7ZL7V3MVLDXQ, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.327699914 (524h44m33.198782949s) [noise=true , address=45.91.116.12:4258 , piece_id=3DFZTINQGUUTX3YDEXERPDRZNN5CKAWOMVAQHL23ES2U5YHH2RAA, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.327720574 (524h44m33.198803609s) [noise=true , address=145.79.227.34:4914 , piece_id=J7JR3XUPHJI3K4TZ23NECWAGL6YLFURO6MB7QR64L7LCEIBMWUXA, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.328318352 (524h44m33.199401387s) [noise=true , address=77.90.31.16:4746 , piece_id=UJJK5KRSEB3JCBXVUSE4TDHJ5JMIEDIGT4EWETKJCBEE46HSM2VA, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.328412561 (524h44m33.199495616s) [noise=true , address=145.79.227.30:4775 , piece_id=D7SJXV4WU5ZGJVYZHZ2UOVAF72ANYWSOCS5BH5U3Z3EI7MXW536A, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.328792936 (524h44m33.199875981s) [noise=true , address=77.90.31.11:4558 , piece_id=HGCZHZGYN6OPOFJJEIMQG2LXMGA6JDHIDDN3RTRF7XSZ56OL3M7Q, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.329056156 (524h44m33.200139191s) [noise=true , address=77.90.31.21:4905 , piece_id=65WTA5P6FPBVAHUJEAX3TIFC23M3A52IZ6FLARKO5RPSSIBZEXWQ, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.329239847 (524h44m33.200322892s) [noise=true , address=64.49.13.169:4709 , piece_id=XOM2255ZNKHB3I26XEYPYVMPCX4QTYGKL6NLXQOPOZ4PQCELVEAQ, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.329737634 (524h44m33.20082068s) [noise=true , address=145.79.227.40:4072 , piece_id=56RK7LN3GVLMGIYZD6FQ2FU23ADTGNWUYMJUMIHH5HHKATLYOQWQ, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.330431346 (524h44m33.201514391s) [noise=true , address=45.91.116.9:4287 , piece_id=F3TDAQGQWDLWZL2WSKWKMMB77TCNBHLIZVY6NPM5MX2ARUSVW37Q, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:34.611654557 (524h44m34.482737602s) [upload=460306]: Next returned nil segment Dec 27 12:13:34.611669636 (524h44m34.482752681s) [upload=460306]: Waiting for error group managing segments... Dec 27 12:13:34.940178727 (524h44m34.811261762s) [noise=true , address=64.49.13.138:4484 , piece_id=4US2ZAFTL4NOMWPTAV6ABOM4GE673PJHVJ7C5B3YGYDVFQUPQLXA, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.218550166 (524h44m35.089633211s) [noise=true , address=145.79.227.32:4860 , piece_id=DXII7EZHDNAX6XTRBF6T2N2LTNP5H3DPHQEGDWKN7ZL7V3MVLDXQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.226012146 (524h44m35.097095191s) [noise=true , address=145.79.227.34:4914 , piece_id=J7JR3XUPHJI3K4TZ23NECWAGL6YLFURO6MB7QR64L7LCEIBMWUXA, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.400566879 (524h44m35.271649914s) [noise=true , address=145.79.227.33:4891 , piece_id=2R47TJ3SLIMPLH4JZTSINHBZXQ3JJKXVETIMILFATRXEI6DZLJLQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.404347268 (524h44m35.275430303s) [noise=true , address=77.90.31.21:4905 , piece_id=65WTA5P6FPBVAHUJEAX3TIFC23M3A52IZ6FLARKO5RPSSIBZEXWQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.416074639 (524h44m35.287157674s) [noise=true , address=77.90.31.13:4644 , piece_id=EP2667PIQIHUWPTWD4REAWTQFSC3VR7V7VZQ344XJH4ASSJP4EMA, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.452194618 (524h44m35.323277663s) [noise=true , address=45.91.116.9:4287 , piece_id=F3TDAQGQWDLWZL2WSKWKMMB77TCNBHLIZVY6NPM5MX2ARUSVW37Q, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.525559618 (524h44m35.396642663s) [noise=true , address=45.91.116.12:4258 , piece_id=3DFZTINQGUUTX3YDEXERPDRZNN5CKAWOMVAQHL23ES2U5YHH2RAA, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.710902755 (524h44m35.58198579s) [noise=true , address=77.90.31.12:4603 , piece_id=Y2CDTLJ5S6DCNVTSXNOYT6FU7TSJYREG2BTIDPMOZAQ6FDXHQYAA, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.833345128 (524h44m35.704428173s) [noise=true , address=77.90.31.15:4721 , piece_id=3A72PLJ5CH6JGOMDL4PUYK3DVIUXRDHVQQA3WA2RW77FXKY46YOQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:36.102928202 (524h44m35.974011237s) [noise=true , address=145.79.227.44:4213 , piece_id=LE27IRRSIIE7T7JQLEMMW4DNJ3XMSRFOPE2DF2XV5XCNWZU3HDEQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:36.153147128 (524h44m36.024230183s) [noise=true , address=145.79.227.40:4072 , piece_id=56RK7LN3GVLMGIYZD6FQ2FU23ADTGNWUYMJUMIHH5HHKATLYOQWQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:36.208908849 (524h44m36.079991884s) [noise=true , address=145.79.227.30:4775 , piece_id=D7SJXV4WU5ZGJVYZHZ2UOVAF72ANYWSOCS5BH5U3Z3EI7MXW536A, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:36.410529180 (524h44m36.281612215s) [noise=true , address=45.91.116.12:4248 , piece_id=G5SOGWARFYFBXQRSRB5OXZZEKN6BMGN3PQC2RBB7JHN55QUPRXIQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:36.410547656 (524h44m36.281630691s) [seg_pos={0 0}, upload=460306]: Segment reached optimal threshold of 14 pieces. Dec 27 12:13:36.410724162 (524h44m36.281807197s) [noise=true , address=77.90.31.11:4558 , piece_id=HGCZHZGYN6OPOFJJEIMQG2LXMGA6JDHIDDN3RTRF7XSZ56OL3M7Q, seg_pos={0 0}, upload=460306]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mGEP1n1icC2gEkyRVy6BeM4vsrgqYWaQeB5rUtxjrWQjyFhTS): protocol: expected piece hash; context canceled; EOF Dec 27 12:13:36.410730103 (524h44m36.281813138s) [noise=true , address=77.90.31.11:4558 , piece_id=HGCZHZGYN6OPOFJJEIMQG2LXMGA6JDHIDDN3RTRF7XSZ56OL3M7Q, seg_pos={0 0}, upload=460306]: Long tail cancellation due to optimal threshold Dec 27 12:13:36.410916448 (524h44m36.281999483s) [noise=true , address=77.90.31.16:4746 , piece_id=UJJK5KRSEB3JCBXVUSE4TDHJ5JMIEDIGT4EWETKJCBEE46HSM2VA, seg_pos={0 0}, upload=460306]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o): protocol: write tcp 188.166.225.170:42390->77.90.31.16:4746: use of closed network connection; write tcp 188.166.225.170:42390->77.90.31.16:4746: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42390->77.90.31.16:4746: use of closed network connection Dec 27 12:13:36.410922750 (524h44m36.282005785s) [noise=true , address=77.90.31.16:4746 , piece_id=UJJK5KRSEB3JCBXVUSE4TDHJ5JMIEDIGT4EWETKJCBEE46HSM2VA, seg_pos={0 0}, upload=460306]: Long tail cancellation due to optimal threshold Dec 27 12:13:36.411020866 (524h44m36.282103911s) [noise=true , address=193.149.164.143:4699 , piece_id=4A5T3OSNSHNKNXKRWOWVJC222TNAISNRGN3NCXQGVFKNHVDBKXHQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LncNSBhhreg9JTByJPSB7eGUEu88Eyd3jc9pg24VsEjp4FULq): protocol: expected piece hash; context canceled; EOF Dec 27 12:13:36.411036306 (524h44m36.282119341s) [noise=true , address=193.149.164.143:4699 , piece_id=4A5T3OSNSHNKNXKRWOWVJC222TNAISNRGN3NCXQGVFKNHVDBKXHQ, seg_pos={0 0}, upload=460306]: Long tail cancellation due to optimal threshold Dec 27 12:13:36.411086592 (524h44m36.282169637s) [noise=true , address=64.49.13.169:4709 , piece_id=XOM2255ZNKHB3I26XEYPYVMPCX4QTYGKL6NLXQOPOZ4PQCELVEAQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:58550->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:58550->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58550->64.49.13.169:4709: use of closed network connection Dec 27 12:13:36.411093095 (524h44m36.28217613s) [noise=true , address=64.49.13.169:4709 , piece_id=XOM2255ZNKHB3I26XEYPYVMPCX4QTYGKL6NLXQOPOZ4PQCELVEAQ, seg_pos={0 0}, upload=460306]: Long tail cancellation due to optimal threshold Dec 27 12:13:36.411118663 (524h44m36.282201708s) [seg_pos={0 0}, upload=460306]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 12:13:36.411202012 (524h44m36.282285047s) [upload=460306]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 12:13:36.411207102 (524h44m36.282290137s) [upload=460306]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 12:13:36.701956341 (524h44m36.573039386s) [upload=460306]: Done uploading segments... Dec 27 12:13:36.701976790 (524h44m36.573059825s) [upload=460306]: Upload finished. err: <nil> Duration: 46.892732086s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....X.......................................................] duration: 3.480284674s trace: https://jaeger.dev.storj.io/trace/12569321705fed16 FAIL (2025-12-27 12:21:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.480313159s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 6.039769277s trace: https://jaeger.dev.storj.io/trace/578b4df2b1bacee FAIL (2025-12-27 12:21:45 UTC) orbiter warning: context deadline exceeded Duration: 6.039795768s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 2.555274135s trace: https://jaeger.dev.storj.io/trace/6dd289079d7aa430 FAIL (2025-12-27 12:21:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.555300075s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................X.................X........] duration: 50.658980958s trace: https://jaeger.dev.storj.io/trace/3b6d882384466243 FAIL (2025-12-27 12:21:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 13QCH91kRMQjVj2ASsVDcyXQ1aapQRVeUm6MMCGimsR6Ns973h, Piece ID: EP2667PIQIHUWPTWD4REAWTQFSC3VR7V7VZQ344XJH4ASSJP4EMA) context deadline exceeded; piecestore: (Node ID: 126Sou5pnMm2eE2FJpv6WYX2VSqJvZZf9rFFJD6hFYDetRD28zu, Piece ID: Y2CDTLJ5S6DCNVTSXNOYT6FU7TSJYREG2BTIDPMOZAQ6FDXHQYAA) context deadline exceeded; piecestore: (Node ID: 12eLKUg2vTnZMrLaraSPEoGXe3jDVBJYyURdcn4f2gCN5pvvCQ8, Piece ID: 3DFZTINQGUUTX3YDEXERPDRZNN5CKAWOMVAQHL23ES2U5YHH2RAA) context deadline exceeded; piecestore: (Node ID: 1X5A7bLqMEQuaMbf9xZ7PDUD4161sz2QqdM9r59y3bsyPBJhaK, Piece ID: 65WTA5P6FPBVAHUJEAX3TIFC23M3A52IZ6FLARKO5RPSSIBZEXWQ) context deadline exceeded ======LOGS====== 38:4484 , piece_id=4US2ZAFTL4NOMWPTAV6ABOM4GE673PJHVJ7C5B3YGYDVFQUPQLXA, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.326171965 (524h44m33.197255s) [noise=true , address=77.90.31.12:4603 , piece_id=Y2CDTLJ5S6DCNVTSXNOYT6FU7TSJYREG2BTIDPMOZAQ6FDXHQYAA, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.326204638 (524h44m33.197287683s) [noise=true , address=77.90.31.13:4644 , piece_id=EP2667PIQIHUWPTWD4REAWTQFSC3VR7V7VZQ344XJH4ASSJP4EMA, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.326588799 (524h44m33.197671844s) [noise=true , address=145.79.227.44:4213 , piece_id=LE27IRRSIIE7T7JQLEMMW4DNJ3XMSRFOPE2DF2XV5XCNWZU3HDEQ, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.326927453 (524h44m33.198010498s) [noise=true , address=77.90.31.15:4721 , piece_id=3A72PLJ5CH6JGOMDL4PUYK3DVIUXRDHVQQA3WA2RW77FXKY46YOQ, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.326992878 (524h44m33.198075923s) [noise=true , address=145.79.227.32:4860 , piece_id=DXII7EZHDNAX6XTRBF6T2N2LTNP5H3DPHQEGDWKN7ZL7V3MVLDXQ, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.327699914 (524h44m33.198782949s) [noise=true , address=45.91.116.12:4258 , piece_id=3DFZTINQGUUTX3YDEXERPDRZNN5CKAWOMVAQHL23ES2U5YHH2RAA, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.327720574 (524h44m33.198803609s) [noise=true , address=145.79.227.34:4914 , piece_id=J7JR3XUPHJI3K4TZ23NECWAGL6YLFURO6MB7QR64L7LCEIBMWUXA, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.328318352 (524h44m33.199401387s) [noise=true , address=77.90.31.16:4746 , piece_id=UJJK5KRSEB3JCBXVUSE4TDHJ5JMIEDIGT4EWETKJCBEE46HSM2VA, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.328412561 (524h44m33.199495616s) [noise=true , address=145.79.227.30:4775 , piece_id=D7SJXV4WU5ZGJVYZHZ2UOVAF72ANYWSOCS5BH5U3Z3EI7MXW536A, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.328792936 (524h44m33.199875981s) [noise=true , address=77.90.31.11:4558 , piece_id=HGCZHZGYN6OPOFJJEIMQG2LXMGA6JDHIDDN3RTRF7XSZ56OL3M7Q, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.329056156 (524h44m33.200139191s) [noise=true , address=77.90.31.21:4905 , piece_id=65WTA5P6FPBVAHUJEAX3TIFC23M3A52IZ6FLARKO5RPSSIBZEXWQ, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.329239847 (524h44m33.200322892s) [noise=true , address=64.49.13.169:4709 , piece_id=XOM2255ZNKHB3I26XEYPYVMPCX4QTYGKL6NLXQOPOZ4PQCELVEAQ, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.329737634 (524h44m33.20082068s) [noise=true , address=145.79.227.40:4072 , piece_id=56RK7LN3GVLMGIYZD6FQ2FU23ADTGNWUYMJUMIHH5HHKATLYOQWQ, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.330431346 (524h44m33.201514391s) [noise=true , address=45.91.116.9:4287 , piece_id=F3TDAQGQWDLWZL2WSKWKMMB77TCNBHLIZVY6NPM5MX2ARUSVW37Q, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:34.611654557 (524h44m34.482737602s) [upload=460306]: Next returned nil segment Dec 27 12:13:34.611669636 (524h44m34.482752681s) [upload=460306]: Waiting for error group managing segments... Dec 27 12:13:34.940178727 (524h44m34.811261762s) [noise=true , address=64.49.13.138:4484 , piece_id=4US2ZAFTL4NOMWPTAV6ABOM4GE673PJHVJ7C5B3YGYDVFQUPQLXA, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.218550166 (524h44m35.089633211s) [noise=true , address=145.79.227.32:4860 , piece_id=DXII7EZHDNAX6XTRBF6T2N2LTNP5H3DPHQEGDWKN7ZL7V3MVLDXQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.226012146 (524h44m35.097095191s) [noise=true , address=145.79.227.34:4914 , piece_id=J7JR3XUPHJI3K4TZ23NECWAGL6YLFURO6MB7QR64L7LCEIBMWUXA, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.400566879 (524h44m35.271649914s) [noise=true , address=145.79.227.33:4891 , piece_id=2R47TJ3SLIMPLH4JZTSINHBZXQ3JJKXVETIMILFATRXEI6DZLJLQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.404347268 (524h44m35.275430303s) [noise=true , address=77.90.31.21:4905 , piece_id=65WTA5P6FPBVAHUJEAX3TIFC23M3A52IZ6FLARKO5RPSSIBZEXWQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.416074639 (524h44m35.287157674s) [noise=true , address=77.90.31.13:4644 , piece_id=EP2667PIQIHUWPTWD4REAWTQFSC3VR7V7VZQ344XJH4ASSJP4EMA, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.452194618 (524h44m35.323277663s) [noise=true , address=45.91.116.9:4287 , piece_id=F3TDAQGQWDLWZL2WSKWKMMB77TCNBHLIZVY6NPM5MX2ARUSVW37Q, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.525559618 (524h44m35.396642663s) [noise=true , address=45.91.116.12:4258 , piece_id=3DFZTINQGUUTX3YDEXERPDRZNN5CKAWOMVAQHL23ES2U5YHH2RAA, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.710902755 (524h44m35.58198579s) [noise=true , address=77.90.31.12:4603 , piece_id=Y2CDTLJ5S6DCNVTSXNOYT6FU7TSJYREG2BTIDPMOZAQ6FDXHQYAA, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.833345128 (524h44m35.704428173s) [noise=true , address=77.90.31.15:4721 , piece_id=3A72PLJ5CH6JGOMDL4PUYK3DVIUXRDHVQQA3WA2RW77FXKY46YOQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:36.102928202 (524h44m35.974011237s) [noise=true , address=145.79.227.44:4213 , piece_id=LE27IRRSIIE7T7JQLEMMW4DNJ3XMSRFOPE2DF2XV5XCNWZU3HDEQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:36.153147128 (524h44m36.024230183s) [noise=true , address=145.79.227.40:4072 , piece_id=56RK7LN3GVLMGIYZD6FQ2FU23ADTGNWUYMJUMIHH5HHKATLYOQWQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:36.208908849 (524h44m36.079991884s) [noise=true , address=145.79.227.30:4775 , piece_id=D7SJXV4WU5ZGJVYZHZ2UOVAF72ANYWSOCS5BH5U3Z3EI7MXW536A, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:36.410529180 (524h44m36.281612215s) [noise=true , address=45.91.116.12:4248 , piece_id=G5SOGWARFYFBXQRSRB5OXZZEKN6BMGN3PQC2RBB7JHN55QUPRXIQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:36.410547656 (524h44m36.281630691s) [seg_pos={0 0}, upload=460306]: Segment reached optimal threshold of 14 pieces. Dec 27 12:13:36.410724162 (524h44m36.281807197s) [noise=true , address=77.90.31.11:4558 , piece_id=HGCZHZGYN6OPOFJJEIMQG2LXMGA6JDHIDDN3RTRF7XSZ56OL3M7Q, seg_pos={0 0}, upload=460306]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mGEP1n1icC2gEkyRVy6BeM4vsrgqYWaQeB5rUtxjrWQjyFhTS): protocol: expected piece hash; context canceled; EOF Dec 27 12:13:36.410730103 (524h44m36.281813138s) [noise=true , address=77.90.31.11:4558 , piece_id=HGCZHZGYN6OPOFJJEIMQG2LXMGA6JDHIDDN3RTRF7XSZ56OL3M7Q, seg_pos={0 0}, upload=460306]: Long tail cancellation due to optimal threshold Dec 27 12:13:36.410916448 (524h44m36.281999483s) [noise=true , address=77.90.31.16:4746 , piece_id=UJJK5KRSEB3JCBXVUSE4TDHJ5JMIEDIGT4EWETKJCBEE46HSM2VA, seg_pos={0 0}, upload=460306]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o): protocol: write tcp 188.166.225.170:42390->77.90.31.16:4746: use of closed network connection; write tcp 188.166.225.170:42390->77.90.31.16:4746: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42390->77.90.31.16:4746: use of closed network connection Dec 27 12:13:36.410922750 (524h44m36.282005785s) [noise=true , address=77.90.31.16:4746 , piece_id=UJJK5KRSEB3JCBXVUSE4TDHJ5JMIEDIGT4EWETKJCBEE46HSM2VA, seg_pos={0 0}, upload=460306]: Long tail cancellation due to optimal threshold Dec 27 12:13:36.411020866 (524h44m36.282103911s) [noise=true , address=193.149.164.143:4699 , piece_id=4A5T3OSNSHNKNXKRWOWVJC222TNAISNRGN3NCXQGVFKNHVDBKXHQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LncNSBhhreg9JTByJPSB7eGUEu88Eyd3jc9pg24VsEjp4FULq): protocol: expected piece hash; context canceled; EOF Dec 27 12:13:36.411036306 (524h44m36.282119341s) [noise=true , address=193.149.164.143:4699 , piece_id=4A5T3OSNSHNKNXKRWOWVJC222TNAISNRGN3NCXQGVFKNHVDBKXHQ, seg_pos={0 0}, upload=460306]: Long tail cancellation due to optimal threshold Dec 27 12:13:36.411086592 (524h44m36.282169637s) [noise=true , address=64.49.13.169:4709 , piece_id=XOM2255ZNKHB3I26XEYPYVMPCX4QTYGKL6NLXQOPOZ4PQCELVEAQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:58550->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:58550->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58550->64.49.13.169:4709: use of closed network connection Dec 27 12:13:36.411093095 (524h44m36.28217613s) [noise=true , address=64.49.13.169:4709 , piece_id=XOM2255ZNKHB3I26XEYPYVMPCX4QTYGKL6NLXQOPOZ4PQCELVEAQ, seg_pos={0 0}, upload=460306]: Long tail cancellation due to optimal threshold Dec 27 12:13:36.411118663 (524h44m36.282201708s) [seg_pos={0 0}, upload=460306]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 12:13:36.411202012 (524h44m36.282285047s) [upload=460306]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 12:13:36.411207102 (524h44m36.282290137s) [upload=460306]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 12:13:36.701956341 (524h44m36.573039386s) [upload=460306]: Done uploading segments... Dec 27 12:13:36.701976790 (524h44m36.573059825s) [upload=460306]: Upload finished. err: <nil> Duration: 50.658998071s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [..X....XX.X....XX......X.X.X.XX.X....X.X..X..XXXXXXX.X.XX...] duration: 6.049173859s trace: https://jaeger.dev.storj.io/trace/45346e3cdaf041e6 FAIL (2025-12-27 12:20:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: U47OO3LFMZFL2VUHE52O7TSWEFJWW6MI7JSC4VJBFLEI7VKLDHSA) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: IBVH6622RTNHARR65U3J4CNDLF4DLMHYMZNXHCZW3PEH44M2243Q) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: X47XMV6LYEE7UWKNOB5YTNX5POAIYI6Q4IK2YJMNTSGTF4J5AYZA) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: 6X2SFFJGT4KMRKBMNUNPFPB4AWQPXHMDPKBSQ23LXD2DOA23N7YA) context deadline exceeded ======LOGS====== Dec 27 12:18:26.324626758 (524h49m26.195709823s) [upload=460370]: Starting upload Dec 27 12:18:26.324732239 (524h49m26.195815274s) [upload=460370]: Done starting upload Dec 27 12:18:26.324935155 (524h49m26.1960182s) [upload=460370]: Uploading segments... Dec 27 12:18:26.325352971 (524h49m26.196436006s) [upload=460370]: Got next segment. Inline: false Dec 27 12:18:26.325497656 (524h49m26.196580701s) [upload=460370]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 12:18:26.325505962 (524h49m26.196589007s) [upload=460370]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 12:18:26.571715459 (524h49m26.442798504s) [seg_pos={0 0}, upload=460370]: Begin upload segment... Dec 27 12:18:26.571776576 (524h49m26.442859611s) [seg_pos={0 0}, upload=460370]: Done begin upload segment. Dec 27 12:18:26.571820409 (524h49m26.442903444s) [noise=true , address=45.91.116.11:4119 , piece_id=MNZWI7FHGVXTBD7WQ5ZXUNE7EZI6V6Z5ZLXYCUFNWAZSZ7AXOJ5Q, seg_pos={0 0}, upload=460370]: Uploading piece... Dec 27 12:18:26.571912755 (524h49m26.44299581s) [noise=true , address=145.79.227.5:4221 , piece_id=AUZ2MPROSW5GJVEFKAXRB5LMEPDUELZ5SKE3FKKZNB4Z55M45ZWQ, seg_pos={0 0}, upload=460370]: Uploading piece... Dec 27 12:18:26.572370166 (524h49m26.443453201s) [noise=true , address=45.91.116.5:4299 , piece_id=USNG6VFZMXRNV3GIT7RRC3SEBINTNPVUAVQ2KNCXU4H2WNJGIABA, seg_pos={0 0}, upload=460370]: Uploading piece... Dec 27 12:18:26.572665278 (524h49m26.443748313s) [noise=true , address=45.91.116.8:4047 , piece_id=6X2SFFJGT4KMRKBMNUNPFPB4AWQPXHMDPKBSQ23LXD2DOA23N7YA, seg_pos={0 0}, upload=460370]: Uploading piece... Dec 27 12:18:26.572679245 (524h49m26.44376228s) [noise=true , address=45.91.116.13:4154 , piece_id=5D6DDAJLOKXCRB6AJ3LZJCMED6GPOJQXMIDJ6YEBENA4T6A3CR3Q, seg_pos={0 0}, upload=460370]: Uploading piece... Dec 27 12:18:26.572931655 (524h49m26.44401469s) [noise=true , address=145.79.227.11:4730 , piece_id=K2XQT66LDU7HQWOHEYQKPJ23UPHVEVQ2G574A6UPZCBW75MZXZ2A, seg_pos={0 0}, upload=460370]: Uploading piece... Dec 27 12:18:26.573039901 (524h49m26.444122936s) [noise=true , address=45.91.116.14:4189 , piece_id=X47XMV6LYEE7UWKNOB5YTNX5POAIYI6Q4IK2YJMNTSGTF4J5AYZA, seg_pos={0 0}, upload=460370]: Uploading piece... Dec 27 12:18:26.573191309 (524h49m26.444274344s) [noise=true , address=77.90.31.19:4841 , piece_id=IBVH6622RTNHARR65U3J4CNDLF4DLMHYMZNXHCZW3PEH44M2243Q, seg_pos={0 0}, upload=460370]: Uploading piece... Dec 27 12:18:26.573346375 (524h49m26.44442942s) [noise=true , address=145.79.227.47:4265 , piece_id=KXJ2UDYTFAYSC72OSAMKEDFVG5YAXTYLSCO22ZENTPOWJECOB3FQ, seg_pos={0 0}, upload=460370]: Uploading piece... Dec 27 12:18:26.573613975 (524h49m26.44469702s) [noise=true , address=145.79.227.10:4403 , piece_id=BRHDGHHEEGI6OKPPAOOVGPNUNMSQZSVFZY5URAEFD5RRU6OROHSQ, seg_pos={0 0}, upload=460370]: Uploading piece... Dec 27 12:18:26.574008987 (524h49m26.445092032s) [noise=true , address=145.79.227.50:4374 , piece_id=XDE56R5TRNB2XZJEY5AHABT3TSRPDLRZZXNBSSJBB4GNR3PJNQQA, seg_pos={0 0}, upload=460370]: Uploading piece... Dec 27 12:18:26.574356889 (524h49m26.445439934s) [noise=true , address=77.90.31.21:4880 , piece_id=WVQ3OYDM4Q3J4F42CBAEWLK73RZI4Y236ZQJ4OKKQXPZ4ZMJYSIQ, seg_pos={0 0}, upload=460370]: Uploading piece... Dec 27 12:18:26.574717195 (524h49m26.44580024s) [noise=true , address=77.90.31.18:4805 , piece_id=AOPNG4OGBU7W7IPEEM55W356FTLK2EKY53AXVSRIZG5MCNQDVTKQ, seg_pos={0 0}, upload=460370]: Uploading piece... Dec 27 12:18:26.575078934 (524h49m26.446161979s) [noise=true , address=77.90.31.12:4590 , piece_id=U47OO3LFMZFL2VUHE52O7TSWEFJWW6MI7JSC4VJBFLEI7VKLDHSA, seg_pos={0 0}, upload=460370]: Uploading piece... Dec 27 12:18:26.575231665 (524h49m26.4463147s) [noise=true , address=145.79.227.20:4438 , piece_id=M3UCTCB4PGIWYO6BKSISA6E3P6P7VOFSIJMBREVDV5PPRHNIBTEA, seg_pos={0 0}, upload=460370]: Uploading piece... Dec 27 12:18:26.575592491 (524h49m26.446675536s) [noise=true , address=145.79.227.42:4123 , piece_id=UZ73OKJTYQWHU3HDSWX4HRBGW6T653EYI3YIPOSR64DUBB6SZQMA, seg_pos={0 0}, upload=460370]: Uploading piece... Dec 27 12:18:28.292908705 (524h49m28.16399175s) [upload=460370]: Next returned nil segment Dec 27 12:18:28.292923312 (524h49m28.164006347s) [upload=460370]: Waiting for error group managing segments... Dec 27 12:18:28.578557081 (524h49m28.449640116s) [noise=true , address=145.79.227.10:4403 , piece_id=BRHDGHHEEGI6OKPPAOOVGPNUNMSQZSVFZY5URAEFD5RRU6OROHSQ, seg_pos={0 0}, upload=460370]: Done uploading piece. err: <nil> Dec 27 12:18:28.579122137 (524h49m28.450205182s) [noise=true , address=145.79.227.11:4730 , piece_id=K2XQT66LDU7HQWOHEYQKPJ23UPHVEVQ2G574A6UPZCBW75MZXZ2A, seg_pos={0 0}, upload=460370]: Done uploading piece. err: <nil> Dec 27 12:18:28.586026825 (524h49m28.45710986s) [noise=true , address=145.79.227.50:4374 , piece_id=XDE56R5TRNB2XZJEY5AHABT3TSRPDLRZZXNBSSJBB4GNR3PJNQQA, seg_pos={0 0}, upload=460370]: Done uploading piece. err: <nil> Dec 27 12:18:28.588843317 (524h49m28.459926352s) [noise=true , address=145.79.227.5:4221 , piece_id=AUZ2MPROSW5GJVEFKAXRB5LMEPDUELZ5SKE3FKKZNB4Z55M45ZWQ, seg_pos={0 0}, upload=460370]: Done uploading piece. err: <nil> Dec 27 12:18:28.606259251 (524h49m28.477342296s) [noise=true , address=145.79.227.20:4438 , piece_id=M3UCTCB4PGIWYO6BKSISA6E3P6P7VOFSIJMBREVDV5PPRHNIBTEA, seg_pos={0 0}, upload=460370]: Done uploading piece. err: <nil> Dec 27 12:18:28.606426169 (524h49m28.477509214s) [noise=true , address=145.79.227.42:4123 , piece_id=UZ73OKJTYQWHU3HDSWX4HRBGW6T653EYI3YIPOSR64DUBB6SZQMA, seg_pos={0 0}, upload=460370]: Done uploading piece. err: <nil> Dec 27 12:18:28.608734464 (524h49m28.479817499s) [noise=true , address=145.79.227.47:4265 , piece_id=KXJ2UDYTFAYSC72OSAMKEDFVG5YAXTYLSCO22ZENTPOWJECOB3FQ, seg_pos={0 0}, upload=460370]: Done uploading piece. err: <nil> Dec 27 12:18:28.745733385 (524h49m28.61681642s) [noise=true , address=77.90.31.19:4841 , piece_id=IBVH6622RTNHARR65U3J4CNDLF4DLMHYMZNXHCZW3PEH44M2243Q, seg_pos={0 0}, upload=460370]: Done uploading piece. err: <nil> Dec 27 12:18:28.785957820 (524h49m28.657040865s) [noise=true , address=45.91.116.14:4189 , piece_id=X47XMV6LYEE7UWKNOB5YTNX5POAIYI6Q4IK2YJMNTSGTF4J5AYZA, seg_pos={0 0}, upload=460370]: Done uploading piece. err: <nil> Dec 27 12:18:28.838673881 (524h49m28.709756916s) [noise=true , address=45.91.116.11:4119 , piece_id=MNZWI7FHGVXTBD7WQ5ZXUNE7EZI6V6Z5ZLXYCUFNWAZSZ7AXOJ5Q, seg_pos={0 0}, upload=460370]: Done uploading piece. err: <nil> Dec 27 12:18:28.844075997 (524h49m28.715159042s) [noise=true , address=45.91.116.8:4047 , piece_id=6X2SFFJGT4KMRKBMNUNPFPB4AWQPXHMDPKBSQ23LXD2DOA23N7YA, seg_pos={0 0}, upload=460370]: Done uploading piece. err: <nil> Dec 27 12:18:28.909998276 (524h49m28.781081311s) [noise=true , address=45.91.116.5:4299 , piece_id=USNG6VFZMXRNV3GIT7RRC3SEBINTNPVUAVQ2KNCXU4H2WNJGIABA, seg_pos={0 0}, upload=460370]: Done uploading piece. err: <nil> Dec 27 12:18:29.485839150 (524h49m29.356922195s) [noise=true , address=77.90.31.12:4590 , piece_id=U47OO3LFMZFL2VUHE52O7TSWEFJWW6MI7JSC4VJBFLEI7VKLDHSA, seg_pos={0 0}, upload=460370]: Done uploading piece. err: <nil> Dec 27 12:18:29.485865370 (524h49m29.356948405s) [seg_pos={0 0}, upload=460370]: Segment reached optimal threshold of 13 pieces. Dec 27 12:18:29.486310016 (524h49m29.357393052s) [noise=true , address=45.91.116.13:4154 , piece_id=5D6DDAJLOKXCRB6AJ3LZJCMED6GPOJQXMIDJ6YEBENA4T6A3CR3Q, seg_pos={0 0}, upload=460370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: write tcp 188.166.225.170:40796->45.91.116.13:4154: use of closed network connection; write tcp 188.166.225.170:40796->45.91.116.13:4154: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40796->45.91.116.13:4154: use of closed network connection Dec 27 12:18:29.486318693 (524h49m29.357401728s) [noise=true , address=45.91.116.13:4154 , piece_id=5D6DDAJLOKXCRB6AJ3LZJCMED6GPOJQXMIDJ6YEBENA4T6A3CR3Q, seg_pos={0 0}, upload=460370]: Long tail cancellation due to optimal threshold Dec 27 12:18:29.486534655 (524h49m29.3576177s) [noise=true , address=77.90.31.21:4880 , piece_id=WVQ3OYDM4Q3J4F42CBAEWLK73RZI4Y236ZQJ4OKKQXPZ4ZMJYSIQ, seg_pos={0 0}, upload=460370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 27 12:18:29.486566305 (524h49m29.35764935s) [noise=true , address=77.90.31.21:4880 , piece_id=WVQ3OYDM4Q3J4F42CBAEWLK73RZI4Y236ZQJ4OKKQXPZ4ZMJYSIQ, seg_pos={0 0}, upload=460370]: Long tail cancellation due to optimal threshold Dec 27 12:18:29.486685321 (524h49m29.357768356s) [noise=true , address=77.90.31.18:4805 , piece_id=AOPNG4OGBU7W7IPEEM55W356FTLK2EKY53AXVSRIZG5MCNQDVTKQ, seg_pos={0 0}, upload=460370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Dec 27 12:18:29.486694368 (524h49m29.357777413s) [noise=true , address=77.90.31.18:4805 , piece_id=AOPNG4OGBU7W7IPEEM55W356FTLK2EKY53AXVSRIZG5MCNQDVTKQ, seg_pos={0 0}, upload=460370]: Long tail cancellation due to optimal threshold Dec 27 12:18:29.486713145 (524h49m29.35779618s) [seg_pos={0 0}, upload=460370]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 12:18:29.486806141 (524h49m29.357889186s) [upload=460370]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 12:18:29.486811362 (524h49m29.357894407s) [upload=460370]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 12:18:29.777510506 (524h49m29.648593541s) [upload=460370]: Done uploading segments... Dec 27 12:18:29.777542577 (524h49m29.648625612s) [upload=460370]: Upload finished. err: <nil> Duration: 6.049201431s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 2.443187692s trace: https://jaeger.dev.storj.io/trace/6dd289079d7aa430 FAIL (2025-12-27 12:20:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.443216577s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....X......................................................] duration: 3.930122227s trace: https://jaeger.dev.storj.io/trace/12569321705fed16 FAIL (2025-12-27 12:20:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.930152275s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.901611168s trace: https://jaeger.dev.storj.io/trace/578b4df2b1bacee FAIL (2025-12-27 12:20:31 UTC) orbiter warning: context deadline exceeded Duration: 5.901631176s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [X..X....XX.X....XX......X.X.X.XX.X....X.X..X..XXXXXXX.X.XX..] duration: 5.676544372s trace: https://jaeger.dev.storj.io/trace/45346e3cdaf041e6 FAIL (2025-12-27 12:20:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: U47OO3LFMZFL2VUHE52O7TSWEFJWW6MI7JSC4VJBFLEI7VKLDHSA) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: IBVH6622RTNHARR65U3J4CNDLF4DLMHYMZNXHCZW3PEH44M2243Q) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: X47XMV6LYEE7UWKNOB5YTNX5POAIYI6Q4IK2YJMNTSGTF4J5AYZA) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: 6X2SFFJGT4KMRKBMNUNPFPB4AWQPXHMDPKBSQ23LXD2DOA23N7YA) context deadline exceeded ======LOGS====== Dec 27 12:18:26.324626758 (524h49m26.195709823s) [upload=460370]: Starting upload Dec 27 12:18:26.324732239 (524h49m26.195815274s) [upload=460370]: Done starting upload Dec 27 12:18:26.324935155 (524h49m26.1960182s) [upload=460370]: Uploading segments... Dec 27 12:18:26.325352971 (524h49m26.196436006s) [upload=460370]: Got next segment. Inline: false Dec 27 12:18:26.325497656 (524h49m26.196580701s) [upload=460370]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 12:18:26.325505962 (524h49m26.196589007s) [upload=460370]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 12:18:26.571715459 (524h49m26.442798504s) [seg_pos={0 0}, upload=460370]: Begin upload segment... Dec 27 12:18:26.571776576 (524h49m26.442859611s) [seg_pos={0 0}, upload=460370]: Done begin upload segment. Dec 27 12:18:26.571820409 (524h49m26.442903444s) [noise=true , address=45.91.116.11:4119 , piece_id=MNZWI7FHGVXTBD7WQ5ZXUNE7EZI6V6Z5ZLXYCUFNWAZSZ7AXOJ5Q, seg_pos={0 0}, upload=460370]: Uploading piece... Dec 27 12:18:26.571912755 (524h49m26.44299581s) [noise=true , address=145.79.227.5:4221 , piece_id=AUZ2MPROSW5GJVEFKAXRB5LMEPDUELZ5SKE3FKKZNB4Z55M45ZWQ, seg_pos={0 0}, upload=460370]: Uploading piece... Dec 27 12:18:26.572370166 (524h49m26.443453201s) [noise=true , address=45.91.116.5:4299 , piece_id=USNG6VFZMXRNV3GIT7RRC3SEBINTNPVUAVQ2KNCXU4H2WNJGIABA, seg_pos={0 0}, upload=460370]: Uploading piece... Dec 27 12:18:26.572665278 (524h49m26.443748313s) [noise=true , address=45.91.116.8:4047 , piece_id=6X2SFFJGT4KMRKBMNUNPFPB4AWQPXHMDPKBSQ23LXD2DOA23N7YA, seg_pos={0 0}, upload=460370]: Uploading piece... Dec 27 12:18:26.572679245 (524h49m26.44376228s) [noise=true , address=45.91.116.13:4154 , piece_id=5D6DDAJLOKXCRB6AJ3LZJCMED6GPOJQXMIDJ6YEBENA4T6A3CR3Q, seg_pos={0 0}, upload=460370]: Uploading piece... Dec 27 12:18:26.572931655 (524h49m26.44401469s) [noise=true , address=145.79.227.11:4730 , piece_id=K2XQT66LDU7HQWOHEYQKPJ23UPHVEVQ2G574A6UPZCBW75MZXZ2A, seg_pos={0 0}, upload=460370]: Uploading piece... Dec 27 12:18:26.573039901 (524h49m26.444122936s) [noise=true , address=45.91.116.14:4189 , piece_id=X47XMV6LYEE7UWKNOB5YTNX5POAIYI6Q4IK2YJMNTSGTF4J5AYZA, seg_pos={0 0}, upload=460370]: Uploading piece... Dec 27 12:18:26.573191309 (524h49m26.444274344s) [noise=true , address=77.90.31.19:4841 , piece_id=IBVH6622RTNHARR65U3J4CNDLF4DLMHYMZNXHCZW3PEH44M2243Q, seg_pos={0 0}, upload=460370]: Uploading piece... Dec 27 12:18:26.573346375 (524h49m26.44442942s) [noise=true , address=145.79.227.47:4265 , piece_id=KXJ2UDYTFAYSC72OSAMKEDFVG5YAXTYLSCO22ZENTPOWJECOB3FQ, seg_pos={0 0}, upload=460370]: Uploading piece... Dec 27 12:18:26.573613975 (524h49m26.44469702s) [noise=true , address=145.79.227.10:4403 , piece_id=BRHDGHHEEGI6OKPPAOOVGPNUNMSQZSVFZY5URAEFD5RRU6OROHSQ, seg_pos={0 0}, upload=460370]: Uploading piece... Dec 27 12:18:26.574008987 (524h49m26.445092032s) [noise=true , address=145.79.227.50:4374 , piece_id=XDE56R5TRNB2XZJEY5AHABT3TSRPDLRZZXNBSSJBB4GNR3PJNQQA, seg_pos={0 0}, upload=460370]: Uploading piece... Dec 27 12:18:26.574356889 (524h49m26.445439934s) [noise=true , address=77.90.31.21:4880 , piece_id=WVQ3OYDM4Q3J4F42CBAEWLK73RZI4Y236ZQJ4OKKQXPZ4ZMJYSIQ, seg_pos={0 0}, upload=460370]: Uploading piece... Dec 27 12:18:26.574717195 (524h49m26.44580024s) [noise=true , address=77.90.31.18:4805 , piece_id=AOPNG4OGBU7W7IPEEM55W356FTLK2EKY53AXVSRIZG5MCNQDVTKQ, seg_pos={0 0}, upload=460370]: Uploading piece... Dec 27 12:18:26.575078934 (524h49m26.446161979s) [noise=true , address=77.90.31.12:4590 , piece_id=U47OO3LFMZFL2VUHE52O7TSWEFJWW6MI7JSC4VJBFLEI7VKLDHSA, seg_pos={0 0}, upload=460370]: Uploading piece... Dec 27 12:18:26.575231665 (524h49m26.4463147s) [noise=true , address=145.79.227.20:4438 , piece_id=M3UCTCB4PGIWYO6BKSISA6E3P6P7VOFSIJMBREVDV5PPRHNIBTEA, seg_pos={0 0}, upload=460370]: Uploading piece... Dec 27 12:18:26.575592491 (524h49m26.446675536s) [noise=true , address=145.79.227.42:4123 , piece_id=UZ73OKJTYQWHU3HDSWX4HRBGW6T653EYI3YIPOSR64DUBB6SZQMA, seg_pos={0 0}, upload=460370]: Uploading piece... Dec 27 12:18:28.292908705 (524h49m28.16399175s) [upload=460370]: Next returned nil segment Dec 27 12:18:28.292923312 (524h49m28.164006347s) [upload=460370]: Waiting for error group managing segments... Dec 27 12:18:28.578557081 (524h49m28.449640116s) [noise=true , address=145.79.227.10:4403 , piece_id=BRHDGHHEEGI6OKPPAOOVGPNUNMSQZSVFZY5URAEFD5RRU6OROHSQ, seg_pos={0 0}, upload=460370]: Done uploading piece. err: <nil> Dec 27 12:18:28.579122137 (524h49m28.450205182s) [noise=true , address=145.79.227.11:4730 , piece_id=K2XQT66LDU7HQWOHEYQKPJ23UPHVEVQ2G574A6UPZCBW75MZXZ2A, seg_pos={0 0}, upload=460370]: Done uploading piece. err: <nil> Dec 27 12:18:28.586026825 (524h49m28.45710986s) [noise=true , address=145.79.227.50:4374 , piece_id=XDE56R5TRNB2XZJEY5AHABT3TSRPDLRZZXNBSSJBB4GNR3PJNQQA, seg_pos={0 0}, upload=460370]: Done uploading piece. err: <nil> Dec 27 12:18:28.588843317 (524h49m28.459926352s) [noise=true , address=145.79.227.5:4221 , piece_id=AUZ2MPROSW5GJVEFKAXRB5LMEPDUELZ5SKE3FKKZNB4Z55M45ZWQ, seg_pos={0 0}, upload=460370]: Done uploading piece. err: <nil> Dec 27 12:18:28.606259251 (524h49m28.477342296s) [noise=true , address=145.79.227.20:4438 , piece_id=M3UCTCB4PGIWYO6BKSISA6E3P6P7VOFSIJMBREVDV5PPRHNIBTEA, seg_pos={0 0}, upload=460370]: Done uploading piece. err: <nil> Dec 27 12:18:28.606426169 (524h49m28.477509214s) [noise=true , address=145.79.227.42:4123 , piece_id=UZ73OKJTYQWHU3HDSWX4HRBGW6T653EYI3YIPOSR64DUBB6SZQMA, seg_pos={0 0}, upload=460370]: Done uploading piece. err: <nil> Dec 27 12:18:28.608734464 (524h49m28.479817499s) [noise=true , address=145.79.227.47:4265 , piece_id=KXJ2UDYTFAYSC72OSAMKEDFVG5YAXTYLSCO22ZENTPOWJECOB3FQ, seg_pos={0 0}, upload=460370]: Done uploading piece. err: <nil> Dec 27 12:18:28.745733385 (524h49m28.61681642s) [noise=true , address=77.90.31.19:4841 , piece_id=IBVH6622RTNHARR65U3J4CNDLF4DLMHYMZNXHCZW3PEH44M2243Q, seg_pos={0 0}, upload=460370]: Done uploading piece. err: <nil> Dec 27 12:18:28.785957820 (524h49m28.657040865s) [noise=true , address=45.91.116.14:4189 , piece_id=X47XMV6LYEE7UWKNOB5YTNX5POAIYI6Q4IK2YJMNTSGTF4J5AYZA, seg_pos={0 0}, upload=460370]: Done uploading piece. err: <nil> Dec 27 12:18:28.838673881 (524h49m28.709756916s) [noise=true , address=45.91.116.11:4119 , piece_id=MNZWI7FHGVXTBD7WQ5ZXUNE7EZI6V6Z5ZLXYCUFNWAZSZ7AXOJ5Q, seg_pos={0 0}, upload=460370]: Done uploading piece. err: <nil> Dec 27 12:18:28.844075997 (524h49m28.715159042s) [noise=true , address=45.91.116.8:4047 , piece_id=6X2SFFJGT4KMRKBMNUNPFPB4AWQPXHMDPKBSQ23LXD2DOA23N7YA, seg_pos={0 0}, upload=460370]: Done uploading piece. err: <nil> Dec 27 12:18:28.909998276 (524h49m28.781081311s) [noise=true , address=45.91.116.5:4299 , piece_id=USNG6VFZMXRNV3GIT7RRC3SEBINTNPVUAVQ2KNCXU4H2WNJGIABA, seg_pos={0 0}, upload=460370]: Done uploading piece. err: <nil> Dec 27 12:18:29.485839150 (524h49m29.356922195s) [noise=true , address=77.90.31.12:4590 , piece_id=U47OO3LFMZFL2VUHE52O7TSWEFJWW6MI7JSC4VJBFLEI7VKLDHSA, seg_pos={0 0}, upload=460370]: Done uploading piece. err: <nil> Dec 27 12:18:29.485865370 (524h49m29.356948405s) [seg_pos={0 0}, upload=460370]: Segment reached optimal threshold of 13 pieces. Dec 27 12:18:29.486310016 (524h49m29.357393052s) [noise=true , address=45.91.116.13:4154 , piece_id=5D6DDAJLOKXCRB6AJ3LZJCMED6GPOJQXMIDJ6YEBENA4T6A3CR3Q, seg_pos={0 0}, upload=460370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: write tcp 188.166.225.170:40796->45.91.116.13:4154: use of closed network connection; write tcp 188.166.225.170:40796->45.91.116.13:4154: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40796->45.91.116.13:4154: use of closed network connection Dec 27 12:18:29.486318693 (524h49m29.357401728s) [noise=true , address=45.91.116.13:4154 , piece_id=5D6DDAJLOKXCRB6AJ3LZJCMED6GPOJQXMIDJ6YEBENA4T6A3CR3Q, seg_pos={0 0}, upload=460370]: Long tail cancellation due to optimal threshold Dec 27 12:18:29.486534655 (524h49m29.3576177s) [noise=true , address=77.90.31.21:4880 , piece_id=WVQ3OYDM4Q3J4F42CBAEWLK73RZI4Y236ZQJ4OKKQXPZ4ZMJYSIQ, seg_pos={0 0}, upload=460370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 27 12:18:29.486566305 (524h49m29.35764935s) [noise=true , address=77.90.31.21:4880 , piece_id=WVQ3OYDM4Q3J4F42CBAEWLK73RZI4Y236ZQJ4OKKQXPZ4ZMJYSIQ, seg_pos={0 0}, upload=460370]: Long tail cancellation due to optimal threshold Dec 27 12:18:29.486685321 (524h49m29.357768356s) [noise=true , address=77.90.31.18:4805 , piece_id=AOPNG4OGBU7W7IPEEM55W356FTLK2EKY53AXVSRIZG5MCNQDVTKQ, seg_pos={0 0}, upload=460370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Dec 27 12:18:29.486694368 (524h49m29.357777413s) [noise=true , address=77.90.31.18:4805 , piece_id=AOPNG4OGBU7W7IPEEM55W356FTLK2EKY53AXVSRIZG5MCNQDVTKQ, seg_pos={0 0}, upload=460370]: Long tail cancellation due to optimal threshold Dec 27 12:18:29.486713145 (524h49m29.35779618s) [seg_pos={0 0}, upload=460370]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 12:18:29.486806141 (524h49m29.357889186s) [upload=460370]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 12:18:29.486811362 (524h49m29.357894407s) [upload=460370]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 12:18:29.777510506 (524h49m29.648593541s) [upload=460370]: Done uploading segments... Dec 27 12:18:29.777542577 (524h49m29.648625612s) [upload=460370]: Upload finished. err: <nil> Duration: 5.676577153s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................X.................X.......] duration: 54.598219289s trace: https://jaeger.dev.storj.io/trace/3b6d882384466243 FAIL (2025-12-27 12:20:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 13QCH91kRMQjVj2ASsVDcyXQ1aapQRVeUm6MMCGimsR6Ns973h, Piece ID: EP2667PIQIHUWPTWD4REAWTQFSC3VR7V7VZQ344XJH4ASSJP4EMA) context deadline exceeded; piecestore: (Node ID: 126Sou5pnMm2eE2FJpv6WYX2VSqJvZZf9rFFJD6hFYDetRD28zu, Piece ID: Y2CDTLJ5S6DCNVTSXNOYT6FU7TSJYREG2BTIDPMOZAQ6FDXHQYAA) context deadline exceeded; piecestore: (Node ID: 12eLKUg2vTnZMrLaraSPEoGXe3jDVBJYyURdcn4f2gCN5pvvCQ8, Piece ID: 3DFZTINQGUUTX3YDEXERPDRZNN5CKAWOMVAQHL23ES2U5YHH2RAA) context deadline exceeded; piecestore: (Node ID: 1X5A7bLqMEQuaMbf9xZ7PDUD4161sz2QqdM9r59y3bsyPBJhaK, Piece ID: 65WTA5P6FPBVAHUJEAX3TIFC23M3A52IZ6FLARKO5RPSSIBZEXWQ) context deadline exceeded ======LOGS====== 38:4484 , piece_id=4US2ZAFTL4NOMWPTAV6ABOM4GE673PJHVJ7C5B3YGYDVFQUPQLXA, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.326171965 (524h44m33.197255s) [noise=true , address=77.90.31.12:4603 , piece_id=Y2CDTLJ5S6DCNVTSXNOYT6FU7TSJYREG2BTIDPMOZAQ6FDXHQYAA, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.326204638 (524h44m33.197287683s) [noise=true , address=77.90.31.13:4644 , piece_id=EP2667PIQIHUWPTWD4REAWTQFSC3VR7V7VZQ344XJH4ASSJP4EMA, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.326588799 (524h44m33.197671844s) [noise=true , address=145.79.227.44:4213 , piece_id=LE27IRRSIIE7T7JQLEMMW4DNJ3XMSRFOPE2DF2XV5XCNWZU3HDEQ, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.326927453 (524h44m33.198010498s) [noise=true , address=77.90.31.15:4721 , piece_id=3A72PLJ5CH6JGOMDL4PUYK3DVIUXRDHVQQA3WA2RW77FXKY46YOQ, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.326992878 (524h44m33.198075923s) [noise=true , address=145.79.227.32:4860 , piece_id=DXII7EZHDNAX6XTRBF6T2N2LTNP5H3DPHQEGDWKN7ZL7V3MVLDXQ, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.327699914 (524h44m33.198782949s) [noise=true , address=45.91.116.12:4258 , piece_id=3DFZTINQGUUTX3YDEXERPDRZNN5CKAWOMVAQHL23ES2U5YHH2RAA, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.327720574 (524h44m33.198803609s) [noise=true , address=145.79.227.34:4914 , piece_id=J7JR3XUPHJI3K4TZ23NECWAGL6YLFURO6MB7QR64L7LCEIBMWUXA, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.328318352 (524h44m33.199401387s) [noise=true , address=77.90.31.16:4746 , piece_id=UJJK5KRSEB3JCBXVUSE4TDHJ5JMIEDIGT4EWETKJCBEE46HSM2VA, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.328412561 (524h44m33.199495616s) [noise=true , address=145.79.227.30:4775 , piece_id=D7SJXV4WU5ZGJVYZHZ2UOVAF72ANYWSOCS5BH5U3Z3EI7MXW536A, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.328792936 (524h44m33.199875981s) [noise=true , address=77.90.31.11:4558 , piece_id=HGCZHZGYN6OPOFJJEIMQG2LXMGA6JDHIDDN3RTRF7XSZ56OL3M7Q, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.329056156 (524h44m33.200139191s) [noise=true , address=77.90.31.21:4905 , piece_id=65WTA5P6FPBVAHUJEAX3TIFC23M3A52IZ6FLARKO5RPSSIBZEXWQ, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.329239847 (524h44m33.200322892s) [noise=true , address=64.49.13.169:4709 , piece_id=XOM2255ZNKHB3I26XEYPYVMPCX4QTYGKL6NLXQOPOZ4PQCELVEAQ, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.329737634 (524h44m33.20082068s) [noise=true , address=145.79.227.40:4072 , piece_id=56RK7LN3GVLMGIYZD6FQ2FU23ADTGNWUYMJUMIHH5HHKATLYOQWQ, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.330431346 (524h44m33.201514391s) [noise=true , address=45.91.116.9:4287 , piece_id=F3TDAQGQWDLWZL2WSKWKMMB77TCNBHLIZVY6NPM5MX2ARUSVW37Q, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:34.611654557 (524h44m34.482737602s) [upload=460306]: Next returned nil segment Dec 27 12:13:34.611669636 (524h44m34.482752681s) [upload=460306]: Waiting for error group managing segments... Dec 27 12:13:34.940178727 (524h44m34.811261762s) [noise=true , address=64.49.13.138:4484 , piece_id=4US2ZAFTL4NOMWPTAV6ABOM4GE673PJHVJ7C5B3YGYDVFQUPQLXA, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.218550166 (524h44m35.089633211s) [noise=true , address=145.79.227.32:4860 , piece_id=DXII7EZHDNAX6XTRBF6T2N2LTNP5H3DPHQEGDWKN7ZL7V3MVLDXQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.226012146 (524h44m35.097095191s) [noise=true , address=145.79.227.34:4914 , piece_id=J7JR3XUPHJI3K4TZ23NECWAGL6YLFURO6MB7QR64L7LCEIBMWUXA, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.400566879 (524h44m35.271649914s) [noise=true , address=145.79.227.33:4891 , piece_id=2R47TJ3SLIMPLH4JZTSINHBZXQ3JJKXVETIMILFATRXEI6DZLJLQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.404347268 (524h44m35.275430303s) [noise=true , address=77.90.31.21:4905 , piece_id=65WTA5P6FPBVAHUJEAX3TIFC23M3A52IZ6FLARKO5RPSSIBZEXWQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.416074639 (524h44m35.287157674s) [noise=true , address=77.90.31.13:4644 , piece_id=EP2667PIQIHUWPTWD4REAWTQFSC3VR7V7VZQ344XJH4ASSJP4EMA, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.452194618 (524h44m35.323277663s) [noise=true , address=45.91.116.9:4287 , piece_id=F3TDAQGQWDLWZL2WSKWKMMB77TCNBHLIZVY6NPM5MX2ARUSVW37Q, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.525559618 (524h44m35.396642663s) [noise=true , address=45.91.116.12:4258 , piece_id=3DFZTINQGUUTX3YDEXERPDRZNN5CKAWOMVAQHL23ES2U5YHH2RAA, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.710902755 (524h44m35.58198579s) [noise=true , address=77.90.31.12:4603 , piece_id=Y2CDTLJ5S6DCNVTSXNOYT6FU7TSJYREG2BTIDPMOZAQ6FDXHQYAA, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.833345128 (524h44m35.704428173s) [noise=true , address=77.90.31.15:4721 , piece_id=3A72PLJ5CH6JGOMDL4PUYK3DVIUXRDHVQQA3WA2RW77FXKY46YOQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:36.102928202 (524h44m35.974011237s) [noise=true , address=145.79.227.44:4213 , piece_id=LE27IRRSIIE7T7JQLEMMW4DNJ3XMSRFOPE2DF2XV5XCNWZU3HDEQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:36.153147128 (524h44m36.024230183s) [noise=true , address=145.79.227.40:4072 , piece_id=56RK7LN3GVLMGIYZD6FQ2FU23ADTGNWUYMJUMIHH5HHKATLYOQWQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:36.208908849 (524h44m36.079991884s) [noise=true , address=145.79.227.30:4775 , piece_id=D7SJXV4WU5ZGJVYZHZ2UOVAF72ANYWSOCS5BH5U3Z3EI7MXW536A, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:36.410529180 (524h44m36.281612215s) [noise=true , address=45.91.116.12:4248 , piece_id=G5SOGWARFYFBXQRSRB5OXZZEKN6BMGN3PQC2RBB7JHN55QUPRXIQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:36.410547656 (524h44m36.281630691s) [seg_pos={0 0}, upload=460306]: Segment reached optimal threshold of 14 pieces. Dec 27 12:13:36.410724162 (524h44m36.281807197s) [noise=true , address=77.90.31.11:4558 , piece_id=HGCZHZGYN6OPOFJJEIMQG2LXMGA6JDHIDDN3RTRF7XSZ56OL3M7Q, seg_pos={0 0}, upload=460306]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mGEP1n1icC2gEkyRVy6BeM4vsrgqYWaQeB5rUtxjrWQjyFhTS): protocol: expected piece hash; context canceled; EOF Dec 27 12:13:36.410730103 (524h44m36.281813138s) [noise=true , address=77.90.31.11:4558 , piece_id=HGCZHZGYN6OPOFJJEIMQG2LXMGA6JDHIDDN3RTRF7XSZ56OL3M7Q, seg_pos={0 0}, upload=460306]: Long tail cancellation due to optimal threshold Dec 27 12:13:36.410916448 (524h44m36.281999483s) [noise=true , address=77.90.31.16:4746 , piece_id=UJJK5KRSEB3JCBXVUSE4TDHJ5JMIEDIGT4EWETKJCBEE46HSM2VA, seg_pos={0 0}, upload=460306]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o): protocol: write tcp 188.166.225.170:42390->77.90.31.16:4746: use of closed network connection; write tcp 188.166.225.170:42390->77.90.31.16:4746: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42390->77.90.31.16:4746: use of closed network connection Dec 27 12:13:36.410922750 (524h44m36.282005785s) [noise=true , address=77.90.31.16:4746 , piece_id=UJJK5KRSEB3JCBXVUSE4TDHJ5JMIEDIGT4EWETKJCBEE46HSM2VA, seg_pos={0 0}, upload=460306]: Long tail cancellation due to optimal threshold Dec 27 12:13:36.411020866 (524h44m36.282103911s) [noise=true , address=193.149.164.143:4699 , piece_id=4A5T3OSNSHNKNXKRWOWVJC222TNAISNRGN3NCXQGVFKNHVDBKXHQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LncNSBhhreg9JTByJPSB7eGUEu88Eyd3jc9pg24VsEjp4FULq): protocol: expected piece hash; context canceled; EOF Dec 27 12:13:36.411036306 (524h44m36.282119341s) [noise=true , address=193.149.164.143:4699 , piece_id=4A5T3OSNSHNKNXKRWOWVJC222TNAISNRGN3NCXQGVFKNHVDBKXHQ, seg_pos={0 0}, upload=460306]: Long tail cancellation due to optimal threshold Dec 27 12:13:36.411086592 (524h44m36.282169637s) [noise=true , address=64.49.13.169:4709 , piece_id=XOM2255ZNKHB3I26XEYPYVMPCX4QTYGKL6NLXQOPOZ4PQCELVEAQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:58550->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:58550->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58550->64.49.13.169:4709: use of closed network connection Dec 27 12:13:36.411093095 (524h44m36.28217613s) [noise=true , address=64.49.13.169:4709 , piece_id=XOM2255ZNKHB3I26XEYPYVMPCX4QTYGKL6NLXQOPOZ4PQCELVEAQ, seg_pos={0 0}, upload=460306]: Long tail cancellation due to optimal threshold Dec 27 12:13:36.411118663 (524h44m36.282201708s) [seg_pos={0 0}, upload=460306]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 12:13:36.411202012 (524h44m36.282285047s) [upload=460306]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 12:13:36.411207102 (524h44m36.282290137s) [upload=460306]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 12:13:36.701956341 (524h44m36.573039386s) [upload=460306]: Done uploading segments... Dec 27 12:13:36.701976790 (524h44m36.573059825s) [upload=460306]: Upload finished. err: <nil> Duration: 54.598242041s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 2.494763834s trace: https://jaeger.dev.storj.io/trace/6dd289079d7aa430 FAIL (2025-12-27 12:19:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.494788131s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 6.340590604s trace: https://jaeger.dev.storj.io/trace/578b4df2b1bacee FAIL (2025-12-27 12:19:45 UTC) orbiter warning: context deadline exceeded Duration: 6.340617806s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [.X..X....XX.X....XX......X.X.X.XX.X....X.X..X..XXXXXXX.X.XX.] duration: 6.022130299s trace: https://jaeger.dev.storj.io/trace/45346e3cdaf041e6 FAIL (2025-12-27 12:19:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: U47OO3LFMZFL2VUHE52O7TSWEFJWW6MI7JSC4VJBFLEI7VKLDHSA) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: IBVH6622RTNHARR65U3J4CNDLF4DLMHYMZNXHCZW3PEH44M2243Q) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: X47XMV6LYEE7UWKNOB5YTNX5POAIYI6Q4IK2YJMNTSGTF4J5AYZA) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: 6X2SFFJGT4KMRKBMNUNPFPB4AWQPXHMDPKBSQ23LXD2DOA23N7YA) context deadline exceeded ======LOGS====== Dec 27 12:18:26.324626758 (524h49m26.195709823s) [upload=460370]: Starting upload Dec 27 12:18:26.324732239 (524h49m26.195815274s) [upload=460370]: Done starting upload Dec 27 12:18:26.324935155 (524h49m26.1960182s) [upload=460370]: Uploading segments... Dec 27 12:18:26.325352971 (524h49m26.196436006s) [upload=460370]: Got next segment. Inline: false Dec 27 12:18:26.325497656 (524h49m26.196580701s) [upload=460370]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 12:18:26.325505962 (524h49m26.196589007s) [upload=460370]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 12:18:26.571715459 (524h49m26.442798504s) [seg_pos={0 0}, upload=460370]: Begin upload segment... Dec 27 12:18:26.571776576 (524h49m26.442859611s) [seg_pos={0 0}, upload=460370]: Done begin upload segment. Dec 27 12:18:26.571820409 (524h49m26.442903444s) [noise=true , address=45.91.116.11:4119 , piece_id=MNZWI7FHGVXTBD7WQ5ZXUNE7EZI6V6Z5ZLXYCUFNWAZSZ7AXOJ5Q, seg_pos={0 0}, upload=460370]: Uploading piece... Dec 27 12:18:26.571912755 (524h49m26.44299581s) [noise=true , address=145.79.227.5:4221 , piece_id=AUZ2MPROSW5GJVEFKAXRB5LMEPDUELZ5SKE3FKKZNB4Z55M45ZWQ, seg_pos={0 0}, upload=460370]: Uploading piece... Dec 27 12:18:26.572370166 (524h49m26.443453201s) [noise=true , address=45.91.116.5:4299 , piece_id=USNG6VFZMXRNV3GIT7RRC3SEBINTNPVUAVQ2KNCXU4H2WNJGIABA, seg_pos={0 0}, upload=460370]: Uploading piece... Dec 27 12:18:26.572665278 (524h49m26.443748313s) [noise=true , address=45.91.116.8:4047 , piece_id=6X2SFFJGT4KMRKBMNUNPFPB4AWQPXHMDPKBSQ23LXD2DOA23N7YA, seg_pos={0 0}, upload=460370]: Uploading piece... Dec 27 12:18:26.572679245 (524h49m26.44376228s) [noise=true , address=45.91.116.13:4154 , piece_id=5D6DDAJLOKXCRB6AJ3LZJCMED6GPOJQXMIDJ6YEBENA4T6A3CR3Q, seg_pos={0 0}, upload=460370]: Uploading piece... Dec 27 12:18:26.572931655 (524h49m26.44401469s) [noise=true , address=145.79.227.11:4730 , piece_id=K2XQT66LDU7HQWOHEYQKPJ23UPHVEVQ2G574A6UPZCBW75MZXZ2A, seg_pos={0 0}, upload=460370]: Uploading piece... Dec 27 12:18:26.573039901 (524h49m26.444122936s) [noise=true , address=45.91.116.14:4189 , piece_id=X47XMV6LYEE7UWKNOB5YTNX5POAIYI6Q4IK2YJMNTSGTF4J5AYZA, seg_pos={0 0}, upload=460370]: Uploading piece... Dec 27 12:18:26.573191309 (524h49m26.444274344s) [noise=true , address=77.90.31.19:4841 , piece_id=IBVH6622RTNHARR65U3J4CNDLF4DLMHYMZNXHCZW3PEH44M2243Q, seg_pos={0 0}, upload=460370]: Uploading piece... Dec 27 12:18:26.573346375 (524h49m26.44442942s) [noise=true , address=145.79.227.47:4265 , piece_id=KXJ2UDYTFAYSC72OSAMKEDFVG5YAXTYLSCO22ZENTPOWJECOB3FQ, seg_pos={0 0}, upload=460370]: Uploading piece... Dec 27 12:18:26.573613975 (524h49m26.44469702s) [noise=true , address=145.79.227.10:4403 , piece_id=BRHDGHHEEGI6OKPPAOOVGPNUNMSQZSVFZY5URAEFD5RRU6OROHSQ, seg_pos={0 0}, upload=460370]: Uploading piece... Dec 27 12:18:26.574008987 (524h49m26.445092032s) [noise=true , address=145.79.227.50:4374 , piece_id=XDE56R5TRNB2XZJEY5AHABT3TSRPDLRZZXNBSSJBB4GNR3PJNQQA, seg_pos={0 0}, upload=460370]: Uploading piece... Dec 27 12:18:26.574356889 (524h49m26.445439934s) [noise=true , address=77.90.31.21:4880 , piece_id=WVQ3OYDM4Q3J4F42CBAEWLK73RZI4Y236ZQJ4OKKQXPZ4ZMJYSIQ, seg_pos={0 0}, upload=460370]: Uploading piece... Dec 27 12:18:26.574717195 (524h49m26.44580024s) [noise=true , address=77.90.31.18:4805 , piece_id=AOPNG4OGBU7W7IPEEM55W356FTLK2EKY53AXVSRIZG5MCNQDVTKQ, seg_pos={0 0}, upload=460370]: Uploading piece... Dec 27 12:18:26.575078934 (524h49m26.446161979s) [noise=true , address=77.90.31.12:4590 , piece_id=U47OO3LFMZFL2VUHE52O7TSWEFJWW6MI7JSC4VJBFLEI7VKLDHSA, seg_pos={0 0}, upload=460370]: Uploading piece... Dec 27 12:18:26.575231665 (524h49m26.4463147s) [noise=true , address=145.79.227.20:4438 , piece_id=M3UCTCB4PGIWYO6BKSISA6E3P6P7VOFSIJMBREVDV5PPRHNIBTEA, seg_pos={0 0}, upload=460370]: Uploading piece... Dec 27 12:18:26.575592491 (524h49m26.446675536s) [noise=true , address=145.79.227.42:4123 , piece_id=UZ73OKJTYQWHU3HDSWX4HRBGW6T653EYI3YIPOSR64DUBB6SZQMA, seg_pos={0 0}, upload=460370]: Uploading piece... Dec 27 12:18:28.292908705 (524h49m28.16399175s) [upload=460370]: Next returned nil segment Dec 27 12:18:28.292923312 (524h49m28.164006347s) [upload=460370]: Waiting for error group managing segments... Dec 27 12:18:28.578557081 (524h49m28.449640116s) [noise=true , address=145.79.227.10:4403 , piece_id=BRHDGHHEEGI6OKPPAOOVGPNUNMSQZSVFZY5URAEFD5RRU6OROHSQ, seg_pos={0 0}, upload=460370]: Done uploading piece. err: <nil> Dec 27 12:18:28.579122137 (524h49m28.450205182s) [noise=true , address=145.79.227.11:4730 , piece_id=K2XQT66LDU7HQWOHEYQKPJ23UPHVEVQ2G574A6UPZCBW75MZXZ2A, seg_pos={0 0}, upload=460370]: Done uploading piece. err: <nil> Dec 27 12:18:28.586026825 (524h49m28.45710986s) [noise=true , address=145.79.227.50:4374 , piece_id=XDE56R5TRNB2XZJEY5AHABT3TSRPDLRZZXNBSSJBB4GNR3PJNQQA, seg_pos={0 0}, upload=460370]: Done uploading piece. err: <nil> Dec 27 12:18:28.588843317 (524h49m28.459926352s) [noise=true , address=145.79.227.5:4221 , piece_id=AUZ2MPROSW5GJVEFKAXRB5LMEPDUELZ5SKE3FKKZNB4Z55M45ZWQ, seg_pos={0 0}, upload=460370]: Done uploading piece. err: <nil> Dec 27 12:18:28.606259251 (524h49m28.477342296s) [noise=true , address=145.79.227.20:4438 , piece_id=M3UCTCB4PGIWYO6BKSISA6E3P6P7VOFSIJMBREVDV5PPRHNIBTEA, seg_pos={0 0}, upload=460370]: Done uploading piece. err: <nil> Dec 27 12:18:28.606426169 (524h49m28.477509214s) [noise=true , address=145.79.227.42:4123 , piece_id=UZ73OKJTYQWHU3HDSWX4HRBGW6T653EYI3YIPOSR64DUBB6SZQMA, seg_pos={0 0}, upload=460370]: Done uploading piece. err: <nil> Dec 27 12:18:28.608734464 (524h49m28.479817499s) [noise=true , address=145.79.227.47:4265 , piece_id=KXJ2UDYTFAYSC72OSAMKEDFVG5YAXTYLSCO22ZENTPOWJECOB3FQ, seg_pos={0 0}, upload=460370]: Done uploading piece. err: <nil> Dec 27 12:18:28.745733385 (524h49m28.61681642s) [noise=true , address=77.90.31.19:4841 , piece_id=IBVH6622RTNHARR65U3J4CNDLF4DLMHYMZNXHCZW3PEH44M2243Q, seg_pos={0 0}, upload=460370]: Done uploading piece. err: <nil> Dec 27 12:18:28.785957820 (524h49m28.657040865s) [noise=true , address=45.91.116.14:4189 , piece_id=X47XMV6LYEE7UWKNOB5YTNX5POAIYI6Q4IK2YJMNTSGTF4J5AYZA, seg_pos={0 0}, upload=460370]: Done uploading piece. err: <nil> Dec 27 12:18:28.838673881 (524h49m28.709756916s) [noise=true , address=45.91.116.11:4119 , piece_id=MNZWI7FHGVXTBD7WQ5ZXUNE7EZI6V6Z5ZLXYCUFNWAZSZ7AXOJ5Q, seg_pos={0 0}, upload=460370]: Done uploading piece. err: <nil> Dec 27 12:18:28.844075997 (524h49m28.715159042s) [noise=true , address=45.91.116.8:4047 , piece_id=6X2SFFJGT4KMRKBMNUNPFPB4AWQPXHMDPKBSQ23LXD2DOA23N7YA, seg_pos={0 0}, upload=460370]: Done uploading piece. err: <nil> Dec 27 12:18:28.909998276 (524h49m28.781081311s) [noise=true , address=45.91.116.5:4299 , piece_id=USNG6VFZMXRNV3GIT7RRC3SEBINTNPVUAVQ2KNCXU4H2WNJGIABA, seg_pos={0 0}, upload=460370]: Done uploading piece. err: <nil> Dec 27 12:18:29.485839150 (524h49m29.356922195s) [noise=true , address=77.90.31.12:4590 , piece_id=U47OO3LFMZFL2VUHE52O7TSWEFJWW6MI7JSC4VJBFLEI7VKLDHSA, seg_pos={0 0}, upload=460370]: Done uploading piece. err: <nil> Dec 27 12:18:29.485865370 (524h49m29.356948405s) [seg_pos={0 0}, upload=460370]: Segment reached optimal threshold of 13 pieces. Dec 27 12:18:29.486310016 (524h49m29.357393052s) [noise=true , address=45.91.116.13:4154 , piece_id=5D6DDAJLOKXCRB6AJ3LZJCMED6GPOJQXMIDJ6YEBENA4T6A3CR3Q, seg_pos={0 0}, upload=460370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: write tcp 188.166.225.170:40796->45.91.116.13:4154: use of closed network connection; write tcp 188.166.225.170:40796->45.91.116.13:4154: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40796->45.91.116.13:4154: use of closed network connection Dec 27 12:18:29.486318693 (524h49m29.357401728s) [noise=true , address=45.91.116.13:4154 , piece_id=5D6DDAJLOKXCRB6AJ3LZJCMED6GPOJQXMIDJ6YEBENA4T6A3CR3Q, seg_pos={0 0}, upload=460370]: Long tail cancellation due to optimal threshold Dec 27 12:18:29.486534655 (524h49m29.3576177s) [noise=true , address=77.90.31.21:4880 , piece_id=WVQ3OYDM4Q3J4F42CBAEWLK73RZI4Y236ZQJ4OKKQXPZ4ZMJYSIQ, seg_pos={0 0}, upload=460370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 27 12:18:29.486566305 (524h49m29.35764935s) [noise=true , address=77.90.31.21:4880 , piece_id=WVQ3OYDM4Q3J4F42CBAEWLK73RZI4Y236ZQJ4OKKQXPZ4ZMJYSIQ, seg_pos={0 0}, upload=460370]: Long tail cancellation due to optimal threshold Dec 27 12:18:29.486685321 (524h49m29.357768356s) [noise=true , address=77.90.31.18:4805 , piece_id=AOPNG4OGBU7W7IPEEM55W356FTLK2EKY53AXVSRIZG5MCNQDVTKQ, seg_pos={0 0}, upload=460370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Dec 27 12:18:29.486694368 (524h49m29.357777413s) [noise=true , address=77.90.31.18:4805 , piece_id=AOPNG4OGBU7W7IPEEM55W356FTLK2EKY53AXVSRIZG5MCNQDVTKQ, seg_pos={0 0}, upload=460370]: Long tail cancellation due to optimal threshold Dec 27 12:18:29.486713145 (524h49m29.35779618s) [seg_pos={0 0}, upload=460370]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 12:18:29.486806141 (524h49m29.357889186s) [upload=460370]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 12:18:29.486811362 (524h49m29.357894407s) [upload=460370]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 12:18:29.777510506 (524h49m29.648593541s) [upload=460370]: Done uploading segments... Dec 27 12:18:29.777542577 (524h49m29.648625612s) [upload=460370]: Upload finished. err: <nil> Duration: 6.022163783s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......X.....................................................] duration: 3.365469685s trace: https://jaeger.dev.storj.io/trace/12569321705fed16 FAIL (2025-12-27 12:19:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.365496466s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......X....................................................] duration: 3.519688586s trace: https://jaeger.dev.storj.io/trace/12569321705fed16 FAIL (2025-12-27 12:19:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.519720026s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................X.................X......] duration: 40.645509725s trace: https://jaeger.dev.storj.io/trace/3b6d882384466243 FAIL (2025-12-27 12:19:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 13QCH91kRMQjVj2ASsVDcyXQ1aapQRVeUm6MMCGimsR6Ns973h, Piece ID: EP2667PIQIHUWPTWD4REAWTQFSC3VR7V7VZQ344XJH4ASSJP4EMA) context deadline exceeded; piecestore: (Node ID: 126Sou5pnMm2eE2FJpv6WYX2VSqJvZZf9rFFJD6hFYDetRD28zu, Piece ID: Y2CDTLJ5S6DCNVTSXNOYT6FU7TSJYREG2BTIDPMOZAQ6FDXHQYAA) context deadline exceeded; piecestore: (Node ID: 12eLKUg2vTnZMrLaraSPEoGXe3jDVBJYyURdcn4f2gCN5pvvCQ8, Piece ID: 3DFZTINQGUUTX3YDEXERPDRZNN5CKAWOMVAQHL23ES2U5YHH2RAA) context deadline exceeded; piecestore: (Node ID: 1X5A7bLqMEQuaMbf9xZ7PDUD4161sz2QqdM9r59y3bsyPBJhaK, Piece ID: 65WTA5P6FPBVAHUJEAX3TIFC23M3A52IZ6FLARKO5RPSSIBZEXWQ) context deadline exceeded ======LOGS====== 38:4484 , piece_id=4US2ZAFTL4NOMWPTAV6ABOM4GE673PJHVJ7C5B3YGYDVFQUPQLXA, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.326171965 (524h44m33.197255s) [noise=true , address=77.90.31.12:4603 , piece_id=Y2CDTLJ5S6DCNVTSXNOYT6FU7TSJYREG2BTIDPMOZAQ6FDXHQYAA, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.326204638 (524h44m33.197287683s) [noise=true , address=77.90.31.13:4644 , piece_id=EP2667PIQIHUWPTWD4REAWTQFSC3VR7V7VZQ344XJH4ASSJP4EMA, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.326588799 (524h44m33.197671844s) [noise=true , address=145.79.227.44:4213 , piece_id=LE27IRRSIIE7T7JQLEMMW4DNJ3XMSRFOPE2DF2XV5XCNWZU3HDEQ, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.326927453 (524h44m33.198010498s) [noise=true , address=77.90.31.15:4721 , piece_id=3A72PLJ5CH6JGOMDL4PUYK3DVIUXRDHVQQA3WA2RW77FXKY46YOQ, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.326992878 (524h44m33.198075923s) [noise=true , address=145.79.227.32:4860 , piece_id=DXII7EZHDNAX6XTRBF6T2N2LTNP5H3DPHQEGDWKN7ZL7V3MVLDXQ, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.327699914 (524h44m33.198782949s) [noise=true , address=45.91.116.12:4258 , piece_id=3DFZTINQGUUTX3YDEXERPDRZNN5CKAWOMVAQHL23ES2U5YHH2RAA, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.327720574 (524h44m33.198803609s) [noise=true , address=145.79.227.34:4914 , piece_id=J7JR3XUPHJI3K4TZ23NECWAGL6YLFURO6MB7QR64L7LCEIBMWUXA, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.328318352 (524h44m33.199401387s) [noise=true , address=77.90.31.16:4746 , piece_id=UJJK5KRSEB3JCBXVUSE4TDHJ5JMIEDIGT4EWETKJCBEE46HSM2VA, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.328412561 (524h44m33.199495616s) [noise=true , address=145.79.227.30:4775 , piece_id=D7SJXV4WU5ZGJVYZHZ2UOVAF72ANYWSOCS5BH5U3Z3EI7MXW536A, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.328792936 (524h44m33.199875981s) [noise=true , address=77.90.31.11:4558 , piece_id=HGCZHZGYN6OPOFJJEIMQG2LXMGA6JDHIDDN3RTRF7XSZ56OL3M7Q, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.329056156 (524h44m33.200139191s) [noise=true , address=77.90.31.21:4905 , piece_id=65WTA5P6FPBVAHUJEAX3TIFC23M3A52IZ6FLARKO5RPSSIBZEXWQ, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.329239847 (524h44m33.200322892s) [noise=true , address=64.49.13.169:4709 , piece_id=XOM2255ZNKHB3I26XEYPYVMPCX4QTYGKL6NLXQOPOZ4PQCELVEAQ, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.329737634 (524h44m33.20082068s) [noise=true , address=145.79.227.40:4072 , piece_id=56RK7LN3GVLMGIYZD6FQ2FU23ADTGNWUYMJUMIHH5HHKATLYOQWQ, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:33.330431346 (524h44m33.201514391s) [noise=true , address=45.91.116.9:4287 , piece_id=F3TDAQGQWDLWZL2WSKWKMMB77TCNBHLIZVY6NPM5MX2ARUSVW37Q, seg_pos={0 0}, upload=460306]: Uploading piece... Dec 27 12:13:34.611654557 (524h44m34.482737602s) [upload=460306]: Next returned nil segment Dec 27 12:13:34.611669636 (524h44m34.482752681s) [upload=460306]: Waiting for error group managing segments... Dec 27 12:13:34.940178727 (524h44m34.811261762s) [noise=true , address=64.49.13.138:4484 , piece_id=4US2ZAFTL4NOMWPTAV6ABOM4GE673PJHVJ7C5B3YGYDVFQUPQLXA, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.218550166 (524h44m35.089633211s) [noise=true , address=145.79.227.32:4860 , piece_id=DXII7EZHDNAX6XTRBF6T2N2LTNP5H3DPHQEGDWKN7ZL7V3MVLDXQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.226012146 (524h44m35.097095191s) [noise=true , address=145.79.227.34:4914 , piece_id=J7JR3XUPHJI3K4TZ23NECWAGL6YLFURO6MB7QR64L7LCEIBMWUXA, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.400566879 (524h44m35.271649914s) [noise=true , address=145.79.227.33:4891 , piece_id=2R47TJ3SLIMPLH4JZTSINHBZXQ3JJKXVETIMILFATRXEI6DZLJLQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.404347268 (524h44m35.275430303s) [noise=true , address=77.90.31.21:4905 , piece_id=65WTA5P6FPBVAHUJEAX3TIFC23M3A52IZ6FLARKO5RPSSIBZEXWQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.416074639 (524h44m35.287157674s) [noise=true , address=77.90.31.13:4644 , piece_id=EP2667PIQIHUWPTWD4REAWTQFSC3VR7V7VZQ344XJH4ASSJP4EMA, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.452194618 (524h44m35.323277663s) [noise=true , address=45.91.116.9:4287 , piece_id=F3TDAQGQWDLWZL2WSKWKMMB77TCNBHLIZVY6NPM5MX2ARUSVW37Q, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.525559618 (524h44m35.396642663s) [noise=true , address=45.91.116.12:4258 , piece_id=3DFZTINQGUUTX3YDEXERPDRZNN5CKAWOMVAQHL23ES2U5YHH2RAA, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.710902755 (524h44m35.58198579s) [noise=true , address=77.90.31.12:4603 , piece_id=Y2CDTLJ5S6DCNVTSXNOYT6FU7TSJYREG2BTIDPMOZAQ6FDXHQYAA, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:35.833345128 (524h44m35.704428173s) [noise=true , address=77.90.31.15:4721 , piece_id=3A72PLJ5CH6JGOMDL4PUYK3DVIUXRDHVQQA3WA2RW77FXKY46YOQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:36.102928202 (524h44m35.974011237s) [noise=true , address=145.79.227.44:4213 , piece_id=LE27IRRSIIE7T7JQLEMMW4DNJ3XMSRFOPE2DF2XV5XCNWZU3HDEQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:36.153147128 (524h44m36.024230183s) [noise=true , address=145.79.227.40:4072 , piece_id=56RK7LN3GVLMGIYZD6FQ2FU23ADTGNWUYMJUMIHH5HHKATLYOQWQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:36.208908849 (524h44m36.079991884s) [noise=true , address=145.79.227.30:4775 , piece_id=D7SJXV4WU5ZGJVYZHZ2UOVAF72ANYWSOCS5BH5U3Z3EI7MXW536A, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:36.410529180 (524h44m36.281612215s) [noise=true , address=45.91.116.12:4248 , piece_id=G5SOGWARFYFBXQRSRB5OXZZEKN6BMGN3PQC2RBB7JHN55QUPRXIQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: <nil> Dec 27 12:13:36.410547656 (524h44m36.281630691s) [seg_pos={0 0}, upload=460306]: Segment reached optimal threshold of 14 pieces. Dec 27 12:13:36.410724162 (524h44m36.281807197s) [noise=true , address=77.90.31.11:4558 , piece_id=HGCZHZGYN6OPOFJJEIMQG2LXMGA6JDHIDDN3RTRF7XSZ56OL3M7Q, seg_pos={0 0}, upload=460306]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mGEP1n1icC2gEkyRVy6BeM4vsrgqYWaQeB5rUtxjrWQjyFhTS): protocol: expected piece hash; context canceled; EOF Dec 27 12:13:36.410730103 (524h44m36.281813138s) [noise=true , address=77.90.31.11:4558 , piece_id=HGCZHZGYN6OPOFJJEIMQG2LXMGA6JDHIDDN3RTRF7XSZ56OL3M7Q, seg_pos={0 0}, upload=460306]: Long tail cancellation due to optimal threshold Dec 27 12:13:36.410916448 (524h44m36.281999483s) [noise=true , address=77.90.31.16:4746 , piece_id=UJJK5KRSEB3JCBXVUSE4TDHJ5JMIEDIGT4EWETKJCBEE46HSM2VA, seg_pos={0 0}, upload=460306]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULxdBTceFr8RctqX6MiwYAQ84tpT9NvedPKgUvibfCb4DwK5o): protocol: write tcp 188.166.225.170:42390->77.90.31.16:4746: use of closed network connection; write tcp 188.166.225.170:42390->77.90.31.16:4746: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42390->77.90.31.16:4746: use of closed network connection Dec 27 12:13:36.410922750 (524h44m36.282005785s) [noise=true , address=77.90.31.16:4746 , piece_id=UJJK5KRSEB3JCBXVUSE4TDHJ5JMIEDIGT4EWETKJCBEE46HSM2VA, seg_pos={0 0}, upload=460306]: Long tail cancellation due to optimal threshold Dec 27 12:13:36.411020866 (524h44m36.282103911s) [noise=true , address=193.149.164.143:4699 , piece_id=4A5T3OSNSHNKNXKRWOWVJC222TNAISNRGN3NCXQGVFKNHVDBKXHQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LncNSBhhreg9JTByJPSB7eGUEu88Eyd3jc9pg24VsEjp4FULq): protocol: expected piece hash; context canceled; EOF Dec 27 12:13:36.411036306 (524h44m36.282119341s) [noise=true , address=193.149.164.143:4699 , piece_id=4A5T3OSNSHNKNXKRWOWVJC222TNAISNRGN3NCXQGVFKNHVDBKXHQ, seg_pos={0 0}, upload=460306]: Long tail cancellation due to optimal threshold Dec 27 12:13:36.411086592 (524h44m36.282169637s) [noise=true , address=64.49.13.169:4709 , piece_id=XOM2255ZNKHB3I26XEYPYVMPCX4QTYGKL6NLXQOPOZ4PQCELVEAQ, seg_pos={0 0}, upload=460306]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:58550->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:58550->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58550->64.49.13.169:4709: use of closed network connection Dec 27 12:13:36.411093095 (524h44m36.28217613s) [noise=true , address=64.49.13.169:4709 , piece_id=XOM2255ZNKHB3I26XEYPYVMPCX4QTYGKL6NLXQOPOZ4PQCELVEAQ, seg_pos={0 0}, upload=460306]: Long tail cancellation due to optimal threshold Dec 27 12:13:36.411118663 (524h44m36.282201708s) [seg_pos={0 0}, upload=460306]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 12:13:36.411202012 (524h44m36.282285047s) [upload=460306]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 12:13:36.411207102 (524h44m36.282290137s) [upload=460306]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 12:13:36.701956341 (524h44m36.573039386s) [upload=460306]: Done uploading segments... Dec 27 12:13:36.701976790 (524h44m36.573059825s) [upload=460306]: Upload finished. err: <nil> Duration: 40.645546466s core uplink us-select-1 us1-placement-20

1
...
111
112
113
114
115
...
6695